Response to Reviewer 7H5B

We thank the reviewer for their detailed and thoughtful comments, and appreciate their recognition of LookWhere’s intuitive design and its ability to achieve strong performance while maintaining computational efficiency. The reviewer raises three key concerns, which we address below.

How much information is lost in the global tokens extracted from a low-resolution input compared to their high-resolution counterparts?:
This is a valid concern that we are happy to address. First, we clarify that LookWhere not only reuses global tokens from the low-resolution input but also relies heavily on its selector to determine where to look – i.e., to identify the most relevant high-resolution patches. We believe this selective mechanism is also central to LookWhere’s effectiveness.

As emphasized in the introduction, many vision tasks involve spatial redundancy, where only small portions of the image are essential for accurate prediction. LookWhere exploits this sparsity by selecting only those high-resolution patches that receive high attention from the teacher. Low-attention patches contribute less to the global context and can be safely discarded with minimal impact on task performance.

We completely agree with the reviewer that a direct comparison with the original tokens from a high-resolution input is crucial to demonstrate the effectiveness of our approach. However, we believe this is already addressed through our experiments, which all include DINOv2 as a baseline. DINOv2 receives the full high-resolution input, and since LookWhere distills a DINOv2 teacher, the performance of the latter serves as a natural upper bound for LookWhere. As shown in Tables 1 and 2, LookWhere achieves comparable performance to DINOv2 on ImageNet classification and ADE20k segmentation using only a fraction of the patches, supporting that LookWhere’s carefully designed cost reduction does not incur a significant loss of meaningful context. Given this helpful feedback we will revise our explanation of the teacher baselines to clarify that they offer the full high-resolution comparison without sparse processing.

Furthermore, in some downstream tasks, LookWhere even outperforms DINOv2. For example, in Traffic Sign Recognition (Table 3), it achieves higher accuracy with 20% of the patches, and, in the Billiard Ball task (see our response to reviewer J83A), it nearly matches DINOv2’s performance with only 20% of the patches. We attribute this strong performance to the selector’s ability to focus precisely on task-relevant regions.

LookWhere also maintains competitive performance across other downstream tasks – including fine-grained bird classification (Table 4), aerial scene classification, and clinical diagnosis (see new experiments in our response to reviewer J83A). These results further support that our selection strategy retains the relevant global context, even without full high-resolution input.

To what extent does the proposed distillation framework depend on the specific properties of DINOv2?:
LookWhere relies on DINOv2 – and any other teacher model – in two key ways: to distill where to look via its saliency maps, and what to see via its representations. While many modern models provide high-quality patch representations suitable for training the extractor, our choice of DINOv2 is primarily motivated by its highly semantic and clean attention maps, which allow the selector to identify visually salient regions without supervision from labels or tasks.

Importantly, any teacher model with semantic attention maps could be used effectively within our framework. To test this generality, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:

• Traffic: 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).
• Birds: 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).
• Billiards: 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see “Generalization to More Domains” in our response to reviewer J83A)

These results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We agree with the reviewer that discussing the specific advantages of DINOv2, and the general properties favorable within a teacher would strengthen the paper’s generality, and we will incorporate this discussion into Section 2.3 on Pretraining.

A mechanism for dynamically determining K:
First, we note that – consistent with prior work on token selection using fixed K values – we observe that most instances within a given task can be handled effectively with a similar number of patches. That said, we agree with the reviewer that dynamically selecting K is a promising direction for improving both performance and efficiency, potentially enabling richer adaptive models. Several strategies could support dynamic K selection, such as thresholding the selector map scores or conditioning K on the input image and a desired accuracy or efficiency level.

We explored one approach during the rebuttal period by selecting high-res patches based on logit thresholds rather than top-K selection (our default in the paper). For both methods, we swept values and measured accuracies on ImageNet-1K. We found that both approaches performed similarly with respect to accuracy and speed. For example, using a logit threshold of 0.03 resulted in 79.48% accuracy and 78.35 tokens on average; whereas setting K=80 tokens resulted in 79.72% accuracy. Given the broader relevance of this idea – extending beyond LookWhere to adaptive token selection more generally – we leave a deeper investigation to future work.

[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025.

Response to Reviewer HJW7

We thank the reviewer for their thoughtful, constructive and encouraging feedback. In particular, we appreciate their recognition of our strong experimental results and are grateful for their suggestions for improving our paper, which we address in this rebuttal.

Robustness of LookWhere Using Alternative Teachers:
LookWhere relies on DINOv2 – and any other teacher model – in two key ways: to distill where to look via its saliency maps, and what to see via its representations. While many modern models provide high-quality patch representations suitable for training the extractor, our choice of DINOv2 is primarily motivated by its highly semantic and clean attention maps, which allow the selector to identify visually salient regions without supervision from labels or tasks.

Importantly, any teacher model with semantic attention maps could be used effectively within our framework. To verify LookWhere’s effectiveness using other teachers, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:

• Traffic: 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).
• Birds: 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).
• Billiards: 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see “Generalization to More Domains” in our response to reviewer J83A)

These results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We reiterate our thanks to the reviewer for this insight that broadens the generality of LookWhere; our experiments confirm its effectiveness, regardless of the teacher used.

What prevents finetuning the selector for task-specific scenarios?:
While nothing fundamentally prevents finetuning the selector for task-specific scenarios, at least in theory, its discrete patch selection makes standard gradient descent inapplicable without modifying the (supervised) training scheme or overall model architecture. One possible direction is adapting a differentiable top-k selection method, such as DPS, or using a reinforcement learning approach as in PatchDrop, to enable selector finetuning after pretraining.

Our results suggest the pretrained selector already generalizes well across tasks – evidenced by the selector maps for Traffic Sign Recognition and CUB (Figures 12 and 13), and strong performance on the newly added Billiard Ball task (see our response to reviewer J83A). In such cases, especially with a suitable teacher, we expect limited gains from selector finetuning. Nevertheless, we agree with the reviewer and acknowledge that finetuning the selector could still improve performance and enhance LookWhere’s flexibility, as well as other adaptive selection schemes such as LTRP, which is why we explicitly mention this limitation in our paper. Given our positive results regarding selector generalization, we hope our findings encourage future research into selector specialization through finetuning.

Could the selective mechanism make LookWhere more sensitive or robust to adversarial examples?:
We thank the reviewer for this insightful question and investigated it further during the rebuttal period by evaluating our LookWhere model under a PGD attack. We find that LookWhere significantly increases robustness to adversarial examples compared to DINOv2, often achieving an additional 20+% accuracy. We summarize these results below and will include them in a new subsection focused on robustness (given an additional room from a 10th page).

PGD attacks with epsilon=1/255

• LookWhere-Base
  • 3 steps: 32.1% acc. on ImageNet-Val
  • 5 steps: 27.5% acc. on ImageNet-Val
  • 10 steps: 24.8% acc. on ImageNet-Val
• DINOv2-Base
  • 3 steps: 8.9% acc. on ImageNet-Val
  • 5 steps: 6.5% acc. on ImageNet-Val
  • 10 steps: 5.3% acc. on ImageNet-Val

[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025.

Response to Reviewer HzUw

We thank the reviewer for their thoughtful feedback, including their recognition of our clear writing and well-motivated approach. We reply to address concerns regarding effectiveness without registers, in addition to new experimental results investigating layer initialization and LookWhere’s generalization to other architectures.

Does the effectiveness of distillation exist when the teacher lacks registers?:
We thank the reviewer for this thoughtful question and agree that registers are indeed beneficial for distillation. However, based on experiments conducted during the rebuttal period, we confirm that registers are not essential for LookWhere’s effectiveness. Registers offer two key benefits: (i) they pass global context from the low-resolution image via the selector, and (ii) they typically produce cleaner attention maps with fewer artifacts, which helps the selector learn better maps. However, these benefits are not unique to registers. Global context can also be delivered through mechanisms such as a CLS token or pooling patches, and teachers without registers can still produce semantic attention maps.

To verify LookWhere’s effectiveness in the absence of registers, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:

• Traffic: 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).
• Birds: 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).
• Billiards: 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see "Generalization to More Domains" in our response to reviewer J83A)

These results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We reiterate our thanks to the reviewer for this insight that broadens the generality of LookWhere; our experiments confirm its effectiveness, regardless of whether the teacher has registers.

Selector and Extractor Layer Initialization:
We want to clarify that only the selector is initialized with the first $L_{\text{low}}$ layers from the teacher; the extractor retains all layers. While a similar strategy could be applied to the extractor, our primary goal is to reduce the selector’s depth for improved efficiency. We retain the extractor’s depth to maintain performance while ensuring efficiency by processing fewer tokens.

Following the reviewer’s suggestion, we explored alternative selector initialization schemes during the rebuttal period. Specifically, we trained four ViT-S models for 100 epochs each using a learning rate of 0.0002, and initializing the selector with different sets of teacher layers: {[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11]}. We summarize these results in Tables T1 and T2 below and generally find LookWhere to be robust to layer initialization – especially with larger K values. At smaller values of K, LookWhere benefits slightly from initializing with deeper layers, aligning with the reviewer’s insight. We thank the reviewer for this suggestion and will include these results alongside our other ablations in Figure 7.

Table T1. Layer Initialization ImageNet kNN Accuracy (%) for Varying Number of Patches (K)

Table T2. Layer Initialization ADE kNN Accuracy (%) for Varying Number of Patches (K)

Could the proposed distillation method be applied beyond ViTs?:

Yes, LookWhere can generalize beyond ViTs, though different considerations are needed for the extractor and selector architectures.

The extractor relies on a partitioning of the input, typically into tokens, that enables: i) selection and processing of sparse inputs, and ii) distillation using partial inputs to learn semantic representations. These two requirements are satisfied by patchification of the high-resolution input, regardless of how the selected patches are later processed. For example, both DPS and IPS use CNN-based selectors and explore different extraction methods, such as pooling or shallow transformers. While alternative extractor architectures are viable, we focus on ViTs due to their effectiveness at seamlessly processing sparse inputs.

Regarding 2D structure, we appreciate the reviewer’s concern that selecting the top-k tokens may disrupt spatial coherence. However, we emphasize that the selector is explicitly trained to select the most informative tokens, meaning any dropped tokens are less critical to the image representation. During pretraining, LookWhere’s patch and class-token distillation losses help ensure that full representation of the high-resolution input can be approximated using only sparse high-resolution tokens which is aided by 2D positional embeddings, also helping ensure structure is maintained.

The selector plays a simpler role of approximating the teacher’s saliency map, which we realize through attention. Hence, any architecture capable of image processing is suitable here. To validate this, we conducted additional experiments during the rebuttal period, training two ViT-B models for 100 epochs each: one using a DINOv2-initialized selector and the other using a CNN initialized EfficientNetV2 and summarize kNN results on ImageNet-HR classification and ADE20K segmentation below (akin to Tables 1 and 2 from our paper) using K values of 16 (1.2%), 72 (5.3%), and 128 (9.3%):

• LookWhere w/ EfficientNetV2 Selector
  • ImageNet-HR: 71.84%, 81.18% and 82.48%
  • ADE20K: 49.34%, 58.80% and 62.25%
• LookWhere w/ DINOv2 Selector
  • ImageNet-HR: 66.16%, 74.92% and 78.42%
  • ADE20K: 49.34%, 58.80% and 62.25%

We also finetuned LookWhere with an EfficientNetV2 Selector for 10 epochs on ImageNet and obtained top-1 accuracy of 82.3%. Overall, we confirm EfficientNet comparable performance to DINOv2 as a selector across all these experiments and validate the ability of LookWhere to generalize beyond ViTs. We will add a new subsection referencing these results, and discussing alternative architectures, to our experiments should our paper be accepted.

Figure 2’s demonstration:
We appreciate the reviewer’s concern regarding the clarity of our central figure. We intentionally abstract away the internals of both the selector (including the FFN) and the extractor to simplify the depiction of LookWhere’s computational flow. We aim to avoid tying the diagram to specific implementation details. While our paper focuses on a ViT-based instantiation of LookWhere, the framework is not limited to ViTs. For example, as demonstrated above, the selector can also be implemented using a CNN. We will revise our figure, clarifying that the output of the selector is a high-resolution selector map to ease interpretation of the losses.

ViT-S or -B based model parameters information:
We agree that including parameter counts for all models, including baselines, facilitates fair comparison. DINOv2, DTEM, PiToME, and ATC are all plain ViT-B architectures with 86 million parameters. In addition to a plain ViT-B, LTRP adds a 22-million-parameter ViT-Small ranking model using 108 million parameters in total. In addition to a plain ViT-B, LookWhere adds 3/12 of a ViT-B along with an FFN to realize its selector, using 109.8 million parameters. However, we measured total inference memory usage across all models to be approximately equal.

[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025.

Response to Reviewer J83A

We thank the reviewer for their thoughtful feedback, particularly their recognition of our extensive ablations, carefully chosen baselines, and clear presentation. We reply to clarify the originality of our work and provide additional ablations as requested, focusing on LookWhere’s compute-performance tradeoff and its generalization to more diverse, high-resolution datasets.

Originality of LookWhere:
While we agree that decoupling “where” and “what” in selective processing is not novel, we believe LookWhere introduces a key innovation through its scalable and straightforward joint pretraining of the selector and extractor via distillation; simultaneously learning where to attend from attention and what to extract from representation, unlike all prior work. As the reviewer notes, it is precisely this distillation that enables LookWhere’s simple yet effective losses, promoting stable and efficient training while leveraging the capabilities of powerful modern models.

In the related work section, we explain LookWhere’s relationship to prior token selection methods such as IPS, DPS, and PatchDrop, which also adopt a decoupled approach. However, we emphasize that these methods rely on complex, task-specific, or multi-stage training pipelines. In contrast, LookWhere achieves superior efficiency and performance through a single-pass distillation process that pretrains both the selector and extractor jointly. By leveraging self-supervised representations, we find the selector generalizes across tasks, requiring only the extractor to be finetuned. We appreciate the reviewer’s feedback and will further underline our novelty by this dual distillation in the revised related work section.

Generalization to More Domains:
We thank the reviewer for suggesting an evaluation on more diverse datasets and fully agree that doing so strengthens our generalization claims. During the rebuttal period, we extended our evaluation to three additional datasets. Following our established methodology, we finetuned both LookWhere and DINOv2 on these new downstream tasks. The results continue to demonstrate LookWhere’s strong performance, as summarized below:

• Aerial Image Dataset (AID) [R1] of 10K images for aerial scene classification across 30 classes. Images were resized from 600px to 602px to match DINOv2’s patch size. We used a 50/50 train/test split and finetuned for 30 epochs. We measure top-1 accuracies of 97.4%, 98.0%, and 98.3% using K values of 184 (10%), 369 (20%), and 924 (50%), respectively, which continues to be competitive with DINOv2’s 98.7% when processing all patches.

• NIH Chest X-ray Dataset [R2] of 112K medical images (90K train / 22K test) for multi-label clinical diagnosis across 14 labels. Images were resized from 1024px to 1022px and finetuned for 10 epochs. Following prior work [R3, R4], we measure AUC-ROC scores of 78.7%, 79.6%, and 81.1% using K values of 532 (10%), 1065 (20%), and 2664 (50%), respectively. DINOv2 achieves a comparable AUC-ROC of 82.2% while processing all patches.

• Billiard Ball Dataset [R5] of 18K images (8K train / 10K test) used to evaluate interpatch reasoning. Each image contains 4-8 balls numbered 1-9; the task is to identify the larger number between the left-most and right-most balls. Images were resized from 1000px to 1008px and finetuned for 30 epochs. We measure top-1 accuracies of 97.3%, 97.5%, and 97.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%), respectively. LookWhere achieves nearly the same accuracy as DINOv2 (97.6%) while only processing a fifth of the patches.

LookWhere’s Compute Performance Tradeoff:
We agree that evaluating LookWhere’s ability to trade off compute for performance is important. As the reviewer notes, we already explored this in two settings: segmentation (Table 2) and ImageNet classification (Figure 7d). During this rebuttal, we vary K more extensively on AID, NIH Chest X-ray, Billiard Balls, and CUB (see below) and will plot accuracy-throughput tradeoff curves, similar to Figure 7d, for these experiments in our paper, should it be accepted.

We initially observed no performance gains using our original LookWhere model during finetuning on CUB across a wider K range, beyond the ratio of 0.2 already included in our paper. This lack of improvement is likely due to the model’s pretraining, which sampled 16 - 128 tokens (out of 1369). To explore this further, we pretrained another model during the rebuttal period using a wider range of token samples (256 - 512) to better support larger K during finetuning.

We then finetuned this model on CUB using our standard approach and measured top-1 accuracies of 88.7%, 90.1% and 90.4% using K values of 136 (10%), 273 (20%) and 684 (50%), respectively. This updated model matches DINOv2’s performance of 90.4% using only half the patches, outperforming all baselines except DTEM, and coming within 0.6% of DTEM’s accuracy while reducing FLOPs by 4.42x. These results further support LookWhere’s ability to reduce computation cost with minimal performance loss. We also note that LookWhere’s performance is closely tied to that of the teacher model, and could improve further with more powerful teachers.

Parameters of all Models:
We agree that including parameter counts for all models, including baselines, facilitates fair comparison. DINOv2, DTEM, PiToME, and ATC are all plain ViT-B architectures with 86 million parameters. In addition to a plain ViT-B, LTRP adds a 22-million-parameter ViT-Small ranking model using 108 million parameters in total. In addition to a plain ViT-B, LookWhere adds 3/12 of a ViT-B along with an FFN to realize its selector, using 109.8 million parameters in total. However, we measured total inference memory usage across all models to be approximately equal.

Cumulative FLOPs:
In Figure 5, “cumulative FLOPs” refers to the total FLOPs consumed during training up to a given point. For example, if a forward and backward pass over a batch requires X FLOPs and there are Y batches per epoch, the cumulative FLOPs after the first epoch would be Y * X, after the second epoch 2 * Y * X, and so on. We will provide further clarification in Figure 5’s caption.

Minor typos:
Thank you for pointing this out. We will correct the typo in the final version.

[R1] G.-S. Xia et al., “AID: A benchmark data set for performance evaluation of aerial scene classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3965–3981, 2017.

[R2] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2097–2106.

[R3] A. Kaku, S. Upadhya, and N. Razavian, “Intermediate layers matter in momentum contrastive self supervised learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 24063–24074, 2021.

[R4] E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar, “Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning,” Nature biomedical engineering, vol. 6, no. 12, pp. 1399–1406, 2022.

[R5] J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and T. Unterthiner, “Differentiable patch selection for image recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2351–2360.

I would like to thank the authors for their detailed response, and for conducting additional experiments. As I mentioned in my initial review, my main concern had to do with the compute-performance trade off curves, which the authors address by conducting additional experiments on CUB, on new datasets, and by mentioning that they will include tradeoff curves in all experiments. I find it very encouraging that LookWhere matched DINOv2 performance with half the patches on CUB, so, I am willing to increase my score, though, I have some additional questions:

• What's the actual speed up of LookWhere compared to DINOv2 when processing half the patches on CUB?
• How adaptive baselines compare against LookWhere at the newly added datasets?
• About Fig. 5, I thank the authors for their clarification, an additional question I have, is what's the intervals between the markers, e.g., 1 epoch?