Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials

Thank you for your valuable feedback. We've addressed your key concerns in our response by expanding our experiments to include larger models and comparisons to other methods, clarifying our method's novelty, and providing new visualizations to improve interpretability.

1. Comparion with other linear attention mechanisms

• We conducted additional experiments on both image classification and image generation tasks, comparing our Visual Contrast Attention with several strong linear attention baselines.

• We first benchmarked our method against other efficient attention mechanisms on the ImageNet-1k classification task. All methods were integrated into a DeiT-Tiny backbone with a similar configuration to ensure a fair comparison. As the results summarized in the table below show, our Visual Contrast Attention achieves the highest Top-1 accuracy of 75.5%, surpassing the next best method, Focused Linear Attention, by 1.4%. This performance gain is achieved with a comparable number of parameters and only a marginal increase in FLOPs, highlighting the effectiveness of our two-stage approach in capturing crucial visual information.

  Linear Attention	Params	FLOPs	Top-1 Acc. (%)
  Hydra Attn	5.7 M	1.1 G	68.3
  Efficient Attn	5.7 M	1.1 G	70.2
  Linear Angular Attn	5.7 M	1.1 G	70.8
  Focused Linear Attn	6.1 M	1.1 G	74.1
  Visual Contrast Attn (ours)	6.0 M	1.2 G	75.5

• Furthermore, to evaluate generative capabilities, we replaced the standard attention layers in a DiT-S/2 with our method and the baselines. We measured the Fréchet Inception Distance (FID) on 50,000 generated samples, where a lower score indicates better image quality. The results in the following table again demonstrate the superiority of our method. Visual Contrast Attention achieves the lowest FID score of 62.1, indicating that it generates higher-fidelity images than the other linear attention mechanisms.

  Linear Attention	Params	FLOPs	FID-50k (lower is better)
  Hydra Attn	33.0 M	5.4 G	121.3
  Efficient Attn	33.0 M	5.1 G	86.5
  Linear Angular Attn	33.0 M	5.6 G	78.2
  Focused Linear Attn	33.8 M	5.6 G	64.7
  Visual Contrast Attn (ours)	33.6 M	5.9 G	62.1

• These new experiments provide a clear and compelling demonstration of our method's advantages in a broader context. We have incorporated these results and a detailed discussion into the revised version of our manuscript.

2. Novelty in vision

• We thank the reviewer for this insightful question, as it allows us to clarify the core novelty of our work. While the concept of differential attention originates in the language domain, our contribution is a fundamental architectural redesign of this idea to solve problems unique to vision. The original differential attention is "flat," operating across all token interactions, which is computationally prohibitive for high-resolution images. In contrast, our Visual-Contrast Attention (VCA) introduces a novel two-stage, hierarchical bottleneck architecture: it first uses a small set of learnable "contrast tokens" to distill a compact global summary of the image's most salient information, and then allows the individual patches to query this summary differentially. This "global distillation, local differential query" mechanism is entirely absent in the original work and is specifically engineered to handle the spatial redundancy and quadratic complexity of visual data.

• This architectural innovation allows us to uniquely solve two problems with a single, elegant mechanism. It simultaneously reduces computational complexity from $O(N^2)$ to near-linear and, by forcing information through a contrastive bottleneck, significantly enhances the model's discriminative power, leading to superior performance. Unlike other efficient ViTs that rely on hand-crafted heuristics like local windows (Swin) or striding (PVT), VCA achieves its dual benefit through a more principled, perception-inspired approach. Therefore, the novelty is not in the subtraction operation itself, but in the creation of a sophisticated architecture that intrinsically unifies computational efficiency and representation enhancement, providing a more fundamental and versatile building block for vision transformers.

3. Results on larger model

• We thank the reviewer for the constructive suggestion regarding model scalability. To address this, we have conducted new experiments on DeiT-Base, which shares an identical architecture with ViT-B. As shown below, our method boosts the Top-1 accuracy by a notable +0.6% with negligible parameter overhead and no change in FLOPs, validating our efficiency claims on a larger scale. Due to the limited rebuttal period, we were unable to finish the ViT-L experiments, but we commit to including these results in the final camera-ready revision.

  Model	Params	FLOPs	Top-1 Acc. (%)
  DeiT-Base	86.6 M	17.6 G	81.8
  +ours (VCA)	87.1 M	17.6 G	82.4

• Since our claim in efficiency focus on its low complexity w.r.t token length, we add an addition ablation with large image resolution (224 -> 448) on VCA-DeiT-Small, to match the FLOPs of DeiT-Base. The results below verified our method can use a smaller model to handle high resultion and longer token sequences, which achieving better final results.

4. Final performance and Concerns on vanilla ViT

• Regarding the performance on ImageNet, we respectfully argue that characterizing the performance gain as "limited" may not fully account for the highly competitive nature of the ImageNet benchmark and the strict constraints of our experiments. Specifically, our VCA module achieves a +3.3% Top-1 accuracy gain on DeiT-Tiny, which is a highly significant result. On the heavily optimized ImageNet benchmark, any improvement over 1%, especially when realized with zero additional computational cost (FLOPs), is typically considered a substantial breakthrough. Crucially, this gain is achieved under "plug-and-play" conditions, without any re-tuning of the original models' hyperparameters. This demonstrates that VCA offers a fundamental improvement in representation quality, not just a marginal one, and its consistent effectiveness across diverse architectures (including DeiT, PVT, Swin, and CSwin) strongly supports its generalizability.

• Regarding the missing ablation on vanilla ViT, we chose DeiT as our primary baseline precisely because its network architecture is identical to that of the original Vision Transformer (ViT). The key contribution of DeiT was its significantly more powerful and data-efficient training strategy, which has since established it as the standard and most widely-used baseline in the Vision Transformer research community. Therefore, performing our analysis on DeiT is not only equivalent to testing on the ViT architecture but also ensures our results are directly and fairly comparable to contemporary state-of-the-art methods. This provides a more meaningful and relevant measure of our method's generalizability and practical impact.

5. Visualization of Contrast Maps

• Due to updated NeurIPS policies, we cannot include images in this rebuttal. Instead, we provide a textual description of our visualization analysis, as requested.

• Visualization Method: We visualize the contrast map in the VCA-DeiT-Tiny model, which has 12 attention blocks. For each block, we pick the head with the largest L2-norm of the Stage-I contrast map. We export the positive $\text{A}^+=\text{Softmax}(\boldsymbol{t}_{+}^{(m)} \boldsymbol{k}^{(m) \top} / \sqrt{d})$ and negative $\text{A}^-$ matrices, compute $\Delta = \text{A}^+ - \lambda \text{A}^-$ , average the $n$ rows, reshape to 2-D grid, upsample to the original image size and overlay it as a heatmap on the original image—highlighting patches that VCA block promotes or suppresses.

• Observations: In early blocks (e.g., block 3), the positive map $A^{+}$ tends to activate on coarse object edges, while the negative map $A^{-}$ activates on background textures. The resulting differential map $\Delta$ often exhibits a classic center-surround pattern, separating the foreground subject from its context. In late blocks (e.g., block 10), $A^{+}$ sharpens its focus on semantic parts (e.g., the head and tail of a bird), while $A^{-}$ becomes more diffuse, indicating that distracting correlations have been effectively filtered out.

• Failure Cases: We observed that the contrastive separation in $\Delta$ is less effective in two main scenarios: (i) texture-dominated scenes, such as an image of "leopard skin," where both positive and negative maps activate strongly on the same texture; and (ii) images with very small objects that are smaller than the resolution of our pooled grid (e.g., 7×7), making it difficult to generate distinct contrastive views. In these cases, VCA's benefit is diminished.

We hope this clarifies our contributions and performance benefits, and we await your feedback.

Thank you for the comprehensive and thoughtful rebuttal. I appreciate the authors’ efforts in providing additional clarification, thorough comparisons, and insightful ablation studies. Based on the strengthened evidence and detailed responses, I believe the rebuttal has addressed most of my concerns, and I have decided to raise my score accordingly. I look forward to seeing the ViT-L experimental results included in the final version.

Thank you for your constructive comments and insightful questions. We have addressed your main concerns below by clarifying key experimental settings and notation, adding ablation studies regarding our core design choices, and providing a deeper elaboration on the conceptual basis for our method and potential hybrid architectures.

1. Ablation on the choice of $n=h \times w$

• For image classification task, we ablated the number n of visual-contrast tokens in VCA-DeiT-Tiny by pooling the original 14 × 14 feature map to 5 × 5, 7 × 7 and 9 × 9 grids (n = 25/49/81). Accuracy rises gradually while the parameter and FLOP budgets increase proportionally. We adopt 7 × 7 in the main paper because it is exactly half the original feautre spatial resolution for a 224 × 224 input.

  grid h × w	n	Params	FLOPs	Top-1 Acc.
  5 × 5	25	5.9 M	1.2 G	75.4
  7 × 7 (ours)	49	6.0 M	1.2 G	75.5
  9 × 9	81	6.1 M	1.3 G	75.7

• For image generation task, On VCA-DiT-S/2 we start from a 16 × 16 latent map (for a 256 × 256 noised image) and pool it to 4 × 4, 8 × 8 and 16 × 16. The same trend appears: FID improves as n grows. We keep 8 × 8 in the submission, again corresponding to one-half of the original latent spatial size.

  grid h × w	n	Params	FLOPs	FID-50k ↓
  4 × 4	16	33.2 M	5.6 G	65.5
  8 × 8 (ours)	64	33.6 M	5.9 G	62.1
  16 × 16	256	33.6 M	7.3 G	58.4

• In general, we keep the h=H/2, w=W/2 choice for all the models, even some model have different feature spatial size in different depth.

2. Ablation on Visual Contrast Token Generation

• We conduct ablation studies on VCA-DeiT-Tiny: for MaxPool we directly replace the AvgPool layer with a MaxPool operator; for ConvPool, we replace it with a 3×3 stride-2 Conv2d so that the 14 × 14 input still shrinks to 7 × 7. All training hyperparameters, data augmentations, and evaluation protocols were kept identical. The results are listed below:

  Token generator	Params	FLOPs	Top-1 Acc. (%)
  AvgPool(ours)	6.0M	1.2G	75.5
  MaxPool	6.0M	1.2G	75.2
  ConvPool	10.0M	1.4G	75.6

• Average pooling outperforms max pooling because it preserves information from every patch instead of discarding everything except the single strongest activation. Moreover, its gradients flow through the entire window rather than a lone peak, which makes optimization smoother and more stable. In practice, it raised Top-1 accuracy by 0.3% in our run while costing exactly the same compute and parameters.

• Compared with the 3 × 3 stride-2 ConvPool alternative, AvgPool is far more efficient. ConvPool inflates the backbone by 4 M extra params and 0.2 G additional FLOPs with only 0.1% accuracy gain. Moreover, AvgPool is parameter-free, deterministic and already highly optimised in hardware libraries, whereas ConvPool introduces extra weight initialisation, regularisation choices and kernel-specific latency.

• Hence, AvgPool offers the best trade-off among information preservation, training stability, and computational efficiency, and is adopted as the default visual-contrast token generator.

3. A more detailed explanation on Tabel 3

• To isolate the effect of Stage I (Line 1 in Table 3), we can combine it with a standard attention mechanism in the second stage, replacing the patch-wise differential attention. In this setup, Stage I proceeds exactly as described.

• To isolate the effect of Stage II (Line 2 in Table 3), we can precede it with a simplified, non-contrastive summarization step. In this variant, we still perform pooling on the query tokens $\tilde{\mathbf{q}}^{(m)}$ to generate a single set of mediator tokens $\tilde{\mathbf{t}}^{(m)}$, but we do not create positive and negative versions. This single set of mediator tokens $\mathbf{t}^{(m)}$ attends to the full key and value sequences to produce one intermediate summary value set, $\hat{\mathbf{v}}_{\text{summary}}^{(m)}$, without any subtraction as in Eq.(10). Then, Stage II is applied as described in the paper, but using this non-contrastive summary.

• For comparsion with differential attention, we also use both differential attention in Stage I and Stage II, as is shown in Line 3, Table 3. This is because differential attention could be plug-and-played and replace with vanilla attention.

4. Typos

• We are very grateful to the reviewers for pointing out the typo in our submission.
• In L147, it should be $\{\mathbf{q}_1, \mathbf{k}_1\}$ and $\{\mathbf{q}_2, \mathbf{k}_2\}$
• The "+" and "-" symbol should use the same symbol, thank for your carefulness.
• We have make sure each equation have its number in the revision.

5. Tensor shape

• Good suggestion! We have clarity its shape upon Eq.(9) appear in the revision, appraciate it!

6. Scale factor

• Thank you for your careful review and for pointing this out. You are correct; the scaling factor in Eq.(11) is a typographical error in the manuscript. Our implementation for all experiments correctly used the standard scaling factor of $\sqrt{d/2}$. We apologize for this oversight and will correct the equation in the revision.

• For completeness, we also ran an ablation study on the VCA-DeiT-Tiny model using the incorrect $\sqrt{d}$ factor from the paper. As shown in the table below, this yielded identical performance to using the correct $\sqrt{d/2}$ factor, which suggests our model is robust to this specific hyperparameter in this setting.

  Model	Attention Scaling Factor	Top-1 Acc. (%)
  VCA-DeiT-Tiny	$\sqrt{d/2}$ (Correct, as used in our experiments)	75.5
  VCA-DeiT-Tiny	$\sqrt{d}$ (Typo in paper, tested for ablation)	75.5

7. Reason for both stage are crucial for vision

• This two-stage architecture is crucial because it mirrors the principles of efficient visual processing, first addressing the immense spatial redundancy in images by distilling a global "figure-versus-ground" context map, akin to identifying the forest before the trees. Subsequently, the patch-wise differential stage allows each local region to query this global map and disambiguate its own role with high precision, deciding whether it belongs to the salient subject or the contextual background. This hierarchical strategy of global abstraction followed by local refinement is not only computationally efficient for high-resolution vision tasks but also aligns with how biological vision systems prioritize and process complex scenes.

• Our ablation results also verify the point our made.

8. Results for interleaved structure

• We thank the reviewer for this insightful question. To investigate the performance of a hybrid approach, we conducted two new experiments on the DeiT-Tiny architecture, creating interleaved structures that substitute half of the standard MHSA blocks with our proposed VCA blocks. The results directly address the question of whether it is beneficial to combine both "similarity" and "difference" pattern learning.

• The first experiment, placing VCA in odd-numbered layers and MHSA in even layers, achieved 75.0% accuracy. The second experiment, with the reverse configuration, yielded 74.3% accuracy. As detailed in the table below, both hybrid models underperform our baseline which uses VCA exclusively, even though they have slightly fewer parameters. The all-VCA model remains superior with an accuracy of 75.5%.

  Configuration	Parameters	FLOPs	Top-1 Acc. (%)
  All VCA (Baseline)	6.0M	1.2G	75.5%
  Odd Layers VCA, Even Layers MHSA	5.8M	1.2G	75.0%
  Even Layers VCA, Odd Layers MHSA	5.8M	1.2G	74.3%

• These findings suggest that it is more effective to consistently apply a single, coherent learning strategy. Standard MHSA learns through "similarity," while our VCA is fundamentally designed to learn through "difference." The superior performance of the all-VCA model indicates that committing fully to the "difference" learning paradigm is more advantageous for this architecture than alternating between the two patterns. This reinforces our central claim that the proposed contrastive attention mechanism is a powerful and sufficient replacement for, rather than a supplement to, standard attention.

We hope this clarifies our contributions and performance benefits, and we await your feedback.

Thank you for your constructive comments and important questions. We have addressed your main concerns below by adding practical hardware speed benchmarks, including a comparison against recent methods like FlashAttention, and clarifying our design choices.

1. Ablation on Visual Contrast Token Generation

• We conduct ablation studies on VCA-DeiT-Tiny: for MaxPool we directly replace the AvgPool layer with a MaxPool operator; for ConvPool, we replace it with a 3×3 stride-2 Conv2d so that the 14 × 14 input still shrinks to 7 × 7. All training hyperparameters, data augmentations, and evaluation protocols were kept identical. The results are listed below:

  Token generator	Params	FLOPs	Top-1 Acc. (%)
  AvgPool(ours)	6.0M	1.2G	75.5
  MaxPool	6.0M	1.2G	75.2
  ConvPool	10.0M	1.4G	75.6

• Average pooling outperforms max pooling because it preserves information from every patch instead of discarding everything except the single strongest activation. Moreover, its gradients flow through the entire window rather than a lone peak, which makes optimization smoother and more stable. In practice, it raised Top-1 accuracy by 0.3% in our run while costing exactly the same compute and parameters.

• Compared with the 3 × 3 stride-2 ConvPool alternative, AvgPool is far more efficient. ConvPool inflates the backbone by 4 M extra params and 0.2 G additional FLOPs with only 0.1% accuracy gain. Moreover, AvgPool is parameter-free, deterministic and already highly optimised in hardware libraries, whereas ConvPool introduces extra weight initialisation, regularisation choices and kernel-specific latency.

• Hence, AvgPool offers the best trade-off among information preservation, training stability, and computational efficiency, and is adopted as the default visual-contrast token generator.

2. Wall-clock training/inference benchmarking & FlashAttention

• We have benchmarked our models to provide concrete performance metrics. We measured training latency (for one ImageNet epoch), peak GPU memory per GPU during training, single-image inference latency (batch size = 1), and inference throughput (batch size = 128).

• Crucially, our Visual Contrast Attention (VCA) is fully compatible with optimizations like FlashAttention-2. The core operations in our method (Eq. 9 and Eq. 11 in our submission) can be implemented with F.scaled_dot_product_attention, allowing VCA to inherit the speed and memory benefits of such fused kernels. Therefore, we report metrics for both a vanilla implementation and an implementation using FlashAttention-2 (+FA2).

• Due to its focus on language modeling, implementation complexity, and limited rebuttal time, we have not provide RetNet (Retentive Network: A Successor to Transformer for Large Language Models) results within this period.

• Setup: Training benchmarks were run on 8xA100 GPUs with a global batch size of 512. Inference benchmarks were run on a single A100 GPU with torch.compile() enabled. For inference, we averaged the results over 1000 runs, discarding the first 100 for warmup. For the Diffusion Transformer (DiT), we report the time for a single denoising step (1-NFE) and exclude VAE decoding time.

  Model	Setting	FLOPs (G)	Acc ↑ / FID ↓	Peak Mem (GB)	Training Latency (min/epoch)	Inference Latency (ms/image)	Throughput (img/s)
  DeiT-T	MHSA	1.2	72.2	18.9	4.1	4.2	3734
  DeiT-T	VCA	1.2	75.5	19.2	4.2	4.3	3701
  DeiT-S	MHSA	4.6	79.8	36.9	8.8	4.3	1321
  DeiT-S	VCA	4.5	80.6	37.3	8.9	4.3	1324
  DiT-S/8	MHSA	0.4	153.8	9.8	12.1	7.8	15637
  DiT-S/8	VCA	0.4	146.3	9.9	12.1	7.8	15602
  ---	---	---	---	---	---	---	---
  DeiT-T	MHSA+FA2	1.2	72.2	16.5	3.1	3.3	4499
  DeiT-T	VCA+FA2	1.2	75.5	16.7	3.1	3.3	4494
  DeiT-S	MHSA+FA2	4.6	79.8	31.1	8.6	3.4	1588
  DeiT-S	VCA+FA2	4.5	80.6	31.4	8.7	3.4	1602
  DiT-S/8	MHSA+FA2	0.4	153.8	9.8	12.1	7.6	16075
  DiT-S/8	VCA+FA2	0.4	146.3	9.8	12.1	7.6	16078

• As the results show, VCA delivers significantly better accuracy/FID with virtually no additional computational overhead. For example, VCA boosts DeiT-T's accuracy by 3.3 points while maintaining the same FLOPs and nearly identical latency and throughput. The minor overhead from pooling and normalization is effectively optimized away by torch.compile, and compatibility with FlashAttention-2 ensures VCA matches the speed of a highly optimized baseline.

3. Speed-up Compared to Models with Similar Performance

• A key advantage of VCA is its ability to achieve the performance of larger, more computationally expensive models with a more efficient architecture. When comparing VCA-equipped models to stronger baselines with similar accuracy, the speed-up is substantial.

  Model	Setting	FLOPs(G)	Acc	Throughput (img/s)
  PVT-T	MHSA	1.9	75.1	2770
  DeiT-T	VCA	1.2	75.5	3701 (↑33.6%)
  ---	---	---	---	---
  Swin-T	MHSA	4.5	81.3	1428
  PVT-S	VCA	4.0	82.3	1764 (↑23.5%)
  ---	---	---	---	---
  Swin-S	MHSA	8.7	83.0	952
  PVT-M	VCA	7.2	83.3	1076 (↑13.0%)

• This comparison highlights VCA's efficiency. For instance, VCA-DeiT-T surpasses PVT-T in accuracy while offering a 33.6% increase in throughput on lower FLOPs. This trend holds for larger models, where our VCA-PVT variants outperform stronger Swin Transformer counterparts with 13.0%-23.5% higher throughput.

4. Extension to video and 3D data

• While our experiments in the paper concentrate on static images, the Visual Contrast Attention paradigm is inherently agnostic to the underlying token structure. By replacing the 2D patch pooling with other sparse sampling techniques, one can extend VCA to videos and 3D data with only a few key changes:

• For structured 3D tokens such as videos and dense voxels, 2D pooling could be replaced with adaptive 3D pooling: for video, divide $T$ frames into t segments and average each segment’s $H \times W$ features to yield $t \times h \times w$ tokens per stream, then add fixed spatiotemporal embeddings $E^{+},E^{-} \in \mathbb{R} ^{t \times h \times w \times d}$. For volumetric data, pool along depth, height, and width to compress $D \times H \times W$ into $d \times h \times w$ positions, attach separate volumetric embeddings, and flatten into n×d positive/negative token sets. To handle variable‐length inputs, one can initialize each embedding table to the maximum sequence length in the dataset and then slice or truncate it to match the current sample’s temporal (or spatial) extent.

• For unstructured 3D data like point clouds and meshes, one could sample or coarsen to $n \ll N$ tokens: for point clouds, one could use FPS plus local feature averaging, coarse voxelization, or lightweight clustering to extract n representative features and then duplicate them into positive/negative streams with distinct embeddings; for meshes or graph-structured data, one could apply edge-collapse mesh-coarsening or graph-pooling (e.g. top-k selection, SAGPool) to reduce the vertex/node count to $n$ and replicate their features similarly.

• In summary, VCA’s two‐stage mechanism only requires swapping the 2D spatial pooling for a suitable spatiotemporal or volumetric pooling/sampling module and adding the corresponding positional embeddings. Everything else—positive/negative streams, differential subtraction, RMSNorm scaling, and two‐stage re-projection—remains unchanged. This design preserves global context at linear cost in the video and 3D domains, just as it does for static images.

We hope this clarifies our contributions and performance benefits, and we await your feedback.

Dear reviewer sZxA

Could you kindly spare a moment to review our rebuttal, where we have addressed each of your comments with additional experiments and analyses? Your time and thoughtful feedback are greatly appreciated.

Sincerely, The Authors of Submission 2322

Dear Reviewer,

I hope this message finds you well.

With the discussion period ending in less than two days, we just wanted to gently follow up on our rebuttal. We hope our response has satisfactorily addressed your valuable concerns.

We would be very grateful for your feedback on our clarifications. Please let us know if there is anything else we can address.

Thank you again for your time and invaluable insights.

Sincerely, The Authors of Submission 2322

We sincerely thank the reviewer for the thorough evaluation and constructive suggestions. We are pleased that the reviewer finds our paper well-structured, broadly applicable, and experimentally solid. Below, we address each concern point-by-point.

1. Wall-clock Latency, Peak GPU Memory, and Throughput

   • We have benchmarked our models to provide concrete performance metrics. We measured training latency (for one ImageNet epoch), peak GPU memory per GPU during training, single-image inference latency (batch size = 1), and inference throughput (batch size = 128).

   • Crucially, our Visual Contrast Attention (VCA) is fully compatible with optimizations like FlashAttention-2. The core operations in our method (Eq. 9 and Eq. 11 in our submission) can be implemented with F.scaled_dot_product_attention, allowing VCA to inherit the speed and memory benefits of such fused kernels. Therefore, we report metrics for both a vanilla implementation and an implementation using FlashAttention-2 (+FA2).

   • Due to its focus on language modeling and implementation complexity, we were unable to benchmark Performer within the limited rebuttal period. However, we did test a similar linear attention architecture; please see our response to Reviewer Z3tv for these results.

   • Setup: Training benchmarks were run on 8xA100 GPUs with a global batch size of 512. Inference benchmarks were run on a single A100 GPU with torch.compile() enabled. For inference, we averaged the results over 1000 runs, discarding the first 100 for warmup. For the Diffusion Transformer (DiT), we report the time for a single denoising step (1-NFE) and exclude VAE decoding time.

   Model	Setting	FLOPs (G)	Acc ↑ / FID ↓	Peak Mem (GB)	Training Latency (min/epoch)	Inference Latency (ms/image)	Throughput (img/s)
   DeiT-T	MHSA	1.2	72.2	18.9	4.1	4.2	3734
   DeiT-T	VCA	1.2	75.5	19.2	4.2	4.3	3701
   DeiT-S	MHSA	4.6	79.8	36.9	8.8	4.3	1321
   DeiT-S	VCA	4.5	80.6	37.3	8.9	4.3	1324
   DiT-S/8	MHSA	0.4	153.8	9.8	12.1	7.8	15637
   DiT-S/8	VCA	0.4	146.3	9.9	12.1	7.8	15602
   ---	---	---	---	---	---	---	---
   DeiT-T	MHSA+FA2	1.2	72.2	16.5	3.1	3.3	4499
   DeiT-T	VCA+FA2	1.2	75.5	16.7	3.1	3.3	4494
   DeiT-S	MHSA+FA2	4.6	79.8	31.1	8.6	3.4	1588
   DeiT-S	VCA+FA2	4.5	80.6	31.4	8.7	3.4	1602
   DiT-S/8	MHSA+FA2	0.4	153.8	9.8	12.1	7.6	16075
   DiT-S/8	VCA+FA2	0.4	146.3	9.8	12.1	7.6	16078

   • As the results show, VCA delivers significantly better accuracy/FID with virtually no additional computational overhead. For example, VCA boosts DeiT-T's accuracy by 3.3 points while maintaining the same FLOPs and nearly identical latency and throughput. The minor overhead from pooling and normalization is effectively optimized away by torch.compile, and compatibility with FlashAttention-2 ensures VCA matches the speed of a highly optimized baseline.

2. Speed-up Compared to Models with Similar Performance

   • A key advantage of VCA is its ability to achieve the performance of larger, more computationally expensive models with a more efficient architecture. When comparing VCA-equipped models to stronger baselines with similar accuracy, the speed-up is substantial.

     Model	Setting	FLOPs(G)	Acc	Throughput (img/s)
     PVT-T	MHSA	1.9	75.1	2770
     DeiT-T	VCA	1.2	75.5	3701 (↑33.6%)
     ---	---	---	---	---
     Swin-T	MHSA	4.5	81.3	1428
     PVT-S	VCA	4.0	82.3	1764 (↑23.5%)
     ---	---	---	---	---
     Swin-S	MHSA	8.7	83.0	952
     PVT-M	VCA	7.2	83.3	1076 (↑13.0%)

   • This comparison highlights VCA's efficiency. For instance, VCA-DeiT-T surpasses PVT-T in accuracy while offering a 33.6% increase in throughput on lower FLOPs. This trend holds for larger models, where our VCA-PVT variants outperform stronger Swin Transformer counterparts with 13.0%-23.5% higher throughput.

3. Statistical Significance

   • To address the concern about statistical significance, we trained the VCA-DeiT-Tiny model three times with different random seeds.

     Model	FLOPs(G)	Run	Top-1 Acc.
     VCA-DeiT-T	1.2	1	75.5
     	1.2	2	75.5
     	1.2	3	75.4

   • The results show minimal variance, with a standard deviation of just 0.05. This demonstrates that the performance gains from VCA are consistent and the training process is stable.

4. Visualization of Contrast Maps

   • Due to updated NeurIPS policies, we cannot include images in this rebuttal. Instead, we provide a textual description of our visualization analysis, as requested.

   • Visualization Method: We visualize the contrast map in the VCA-DeiT-Tiny model, which has 12 attention blocks. For each block, we pick the head with the largest L2-norm of the Stage-I contrast map. We export the positive $\text{A}^+=\text{Softmax}(\boldsymbol{t}_{+}^{(m)} \boldsymbol{k}^{(m) \top} / \sqrt{d})$ and negative $\text{A}^-$ matrices, compute $\Delta = \text{A}^+ - \lambda \text{A}^-$ , average the $n$ rows, reshape to 2-D grid, upsample to the original image size and overlay it as a heatmap on the original image—highlighting patches that VCA block promotes or suppresses.

   • Observations: In early blocks (e.g., block 3), the positive map $A^{+}$ tends to activate on coarse object edges, while the negative map $A^{-}$ activates on background textures. The resulting differential map $\Delta$ often exhibits a classic center-surround pattern, separating the foreground subject from its context. In late blocks (e.g., block 10), $A^{+}$ sharpens its focus on semantic parts (e.g., the head and tail of a bird), while $A^{-}$ becomes more diffuse, indicating that distracting correlations have been effectively filtered out.

   • Failure Cases: We observed that the contrastive separation in $\Delta$ is less effective in two main scenarios: (i) texture-dominated scenes, such as an image of "leopard skin," where both positive and negative maps activate strongly on the same texture; and (ii) images with very small objects that are smaller than the resolution of our pooled grid (e.g., 7×7), making it difficult to generate distinct contrastive views. In these cases, VCA's benefit is diminished.

5. Ablation on Pooled Grid Size

   • As requested, we analyzed how performance varies with the number of visual contrast tokens ($n$), which is determined by the pooled grid size. This clarifies the trade-off between computational cost and model fidelity.

   • Image Classification (VCA-DeiT-Tiny): We ablated the number of tokens by pooling the original 14×14 feature map to 5×5, 7×7, and 9×9 grids.

     Grid $h$×$w$	$n$	Params	FLOPs	Top-1 Acc.	Throughput (img/s)
     5×5	25	5.9 M	1.2 G	75.4	4517
     7×7 (ours)	49	6.0 M	1.2 G	75.5	4494
     9×9	81	6.1 M	1.3 G	75.7	4263

   • The results show a clear trade-off: increasing $n$ yields modest accuracy gains at the cost of slightly more parameters and reduced throughput. We chose 7×7 as our default because it offers a strong balance and corresponds to half the spatial resolution of the input feature map.

   • Image Generation (VCA-DiT-S/2): We pooled the 16×16 latent map to 4×4, 8×8, and 16×16.

     Grid $h$×$w$	$n$	Params	FLOPs	FID-50k ↓
     4×4	16	33.2 M	5.6 G	65.5
     8×8 (ours)	64	33.6 M	5.9 G	62.1
     16×16	256	33.6 M	7.3 G	58.4

   • A similar trend appears, where a larger grid improves the FID score. We chose 8×8 for our main experiments, as it again represents half the latent's spatial resolution and provides a compelling trade-off between generation quality and computational cost.

We hope this clarifies our contributions and performance benefits, and we await your feedback.

Dear reviewer f3ww

Could you kindly spare a moment to review our rebuttal, where we have addressed each of your comments with additional experiments and analyses? Your time and thoughtful feedback are greatly appreciated.

Sincerely, The Authors of Submission 2322

Dear Reviewer,

I hope this message finds you well.

With the discussion period ending in less than two days, we just wanted to gently follow up on our rebuttal. We hope our response has satisfactorily addressed your valuable concerns.

We would be very grateful for your feedback on our clarifications. Please let us know if there is anything else we can address.

Thank you again for your time and invaluable insights.

Sincerely, The Authors of Submission 2322

We sincerely thank the reviewer for their time and completely understand that unexpected professional obligations can prevent a review.

• We noticed that while your comment explains the situation, the submission still includes numerical scores and a final rating of "Borderline Reject." We would like to highlight that this stands in contrast to the detailed and very positive assessments from the other reviewers. They described our work as "an excellent work" and "well-structured," praising its "impressive experimental results," "clear logic," and the "significant performance improvement" achieved without additional computational cost.

• Given this, we would be very grateful if your schedule now permits you to take a look at our paper; the strengths identified by others might be of interest to you as well. If a full review is not feasible, we would kindly and respectfully ask if you might consider adjusting the numerical scores to reflect that a detailed assessment was not performed. Our concern is that the current "Borderline Reject" rating, particularly with a confidence score of 1, could have an unintended influence on the final decision, which would not align with the thorough and positive evaluations provided by the other reviewers.

Thank you again for your time and understanding.