We thank the reviewer for the insightful feedback. We (i) extend the experiments, (ii) revisit over-smoothing, and (iii) answer your questions.

---

1. Main concerns

1. Limited evaluation / marginal gains
2. Cosine similarity appears higher — unclear whether over-smoothing is mitigated.
3. Design questions — wave-equation choice, causal attention, Equation 15 options, ablation.

---

2. Extended evaluation

2.1 Citation-graph benchmarks

We adopt DIFFormer (Wu et al. 2023) and add our wave update (τ = 0.2). All hyperparameters remain unchanged. We report mean ± SD over 5 seeds.

Table 1 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)

† Paired t-test (5 matched seeds): p > 0.05 → no statistically significant degradation.

▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.

2.2 ImageNet-1K (CaIT-XXS-24)

Table 2 Comparison with CaIT-XXS-24

Same recipe as CaIT; LayerScale 1e-5, no extra class-attention block.

Implementation note. By omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) and still gains +1 pt Top-1 accuracy.

---

3. Over-smoothing revisited

Table 3 Cosine similarity (Cora, 20 layers)

As depth increases, diffusion saturates to ≈ 1.00, showing severe over-smoothing, whereas the wave-enhanced model stays below 0.38, preserving representational diversity and enabling the large accuracy gains highlighted above.

Table 4 Spectral gap & Node-feature variance (DeiT-Tiny, ImageNet-1K val dataset: 50k images)

Here the spectral gap is defined as $1-\lambda_{2}$, where $\lambda_{2}$ the second-largest eigenvalue (in magnitude) of the right-stochastic attention matrix $A$. A smaller gap, together with higher node-feature variance, indicates reduced over-smoothing beyond what cosine similarity alone captures.

---

4. Ablation on ImageNet-1K (DeiT-Tiny, 150 epochs)

Table 5 Velocity-FFN ablation

---

5. Responses to the reviewer’s questions

---

6. Planned manuscript updates

• Typos & clarity — correct typo and tighten any ambiguous expression in last few paragraphs in section 3.2.
• Related-work – explicitly add a note on causal attention.
• Experiments – incorporate citation-graph benchmark results, layer-wise cosine-similarity trends with results of the spectral gaps and node-feature variances, and the ablation table.
• All training scripts will be released upon acceptance.

---

Remark — Ongoing fine-tuning experiments (GPT-2 / GPT-2-Large)

We are currently fine-tuning GPT-2 (124 M) and may also fine-tune GPT-2-Large (774 M) on the OpenWebText corpus, comparing the diffusive baseline with our wave update. Experiments are ongoing; full results will be reported in the camera-ready.

---

We hope these additions fully address your questions and demonstrate the broader impact of Wavy Transformer. We would appreciate it if you could reconsider the overall score in light of this evidence.

Thank you again for your time and consideration.

Thank you again for your thoughtful feedback. The core Wavy-Transformer method is unchanged; we have only added the empirical evidence and clarifications you requested. Our intent is not to impose a full reread, but simply to let you confirm that each of your stated concerns is now addressed.

Quick Verification Checklist

---

• Marginal gains / limited evaluation
  • Where to look: Table 1 (sparse graph benchmark) & Table 2 (deeper vision model, ImageNet-1K)
  • Evidence: Graphs +20 $\sim$ 45 pt; ImageNet +1 pt Top-1, –7 % params
  • Take-away: Performance gains grow with depth and generalize beyond dense vision/NLP tasks.

---

• CosSim still high $\to$ over-smoothing?
  • Where to look: Table 3 (CosSim) & Table 4 (spectral gap / variance)
  • Evidence: CosSim 1.00 $\to$ 0.38; Gap 0.836 $\to$ 0.629; Variance 2.480 $\to$ 2.609
  • Take-away: Wave updates slow diffusion and keep features diverse, even if angles stay small.

---

• Design & ablation
  • Where to look: Table 5 (velocity-FFN ablation) & Q2–Q4 (design rationale)
  • Evidence: Without FFN –0.53 pt; mini-GPT decoder val-loss 1.470 $\to$ 1.460
  • Take-away: Results validate the mechanism and show that non-diffusive updates remain effective—even with causal attention.

---

Reproducibility: All code and exact training scripts will be released upon acceptance.

Note: GPT-2/L runs are nice-to-have evidence and not part of our acceptance claim. Thus, they will appear in the appendix if accepted as supplementary evidence.

We hope this checklist simplifies verification. If it resolves your earlier concerns, we would be grateful for any score reconsideration you deem appropriate. Thank you again for your time and help in improving the work.

Thank you for the positive assessment and for the clear questions on scale and computational cost.

We (i) add deeper models in vision & graph, (ii) detail runtime/memory, and (iii) answer your points.

---

1. Main concerns

1. Scalability – behavior on deeper / larger models?
2. Runtime & memory – overhead of the extra velocity state?

---

2. Additional large-depth experiments

2.1 ImageNet-1K (CaIT-XXS-24)

Table 1 Comparison with CaIT-XXS-24

Same training recipe as CaIT; LayerScale 1e-5, no extra class-attention block.

Implementation note. By omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) yet achieves a +1 pt Top-1 gain.

2.2 Citation-graph benchmarks

We adopt DIFFormer (Wu et al. 2023) and only add the wave update without adding Velocity-FFN/LN (τ = 0.2). All hyperparameters remain unchanged. We report mean ± SD over 5 seeds.

Table 2 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)

† Paired t-test (5 matched seeds): p > 0.05 → no statistically significant degradation.

▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.

Full results (τ = 0.2 / 0.5) will be included in the final version.

Table 3 Cosine similarity (Cora, 20 layers)

As depth increases, Diffusion saturates to ≈ 1.00, showing severe over-smoothing, whereas the Wave-enhanced model stays below 0.38, preserving representational diversity and enabling the large accuracy gains highlighted above.

---

3. Computational cost (4 × V100) — Light vs. Full Wave

Table 4 Comparison of runtime and memory costs of BERTs

Table 5 Comparison of runtime and memory costs of DeiT-Tiny

Notes

• Full Wave (original proposal).
  Implements a discrete Hamiltonian form of the wave equation by introducing an explicit velocity state plus Velocity-FFN and Velocity-LN. While it raises accuracy on some tasks, the extra state and auxiliary blocks slow both inference and training, as shown above.

• Lighter Wave (second-order, momentum-only).
  Responding to the reviewer’s cost concern, we removed the auxiliary Velocity-FFN/LN and kept only the momentum term

  $$x_{l+1}=x_l+\operatorname{attn}(x_l)+\beta\bigl(x_l-x_{l-1}\bigr),$$

  where $\beta$ is the mixing parameter. This preserves the physical insight, yet matches baseline throughput (NLP -0.30 %, CV +0.5 %) and memory (NLP +2.1 %, CV +10.7 %, still < 1 GB).

• Effect on deep graphs.
  Applying Lighter Wave to 20-layer citation-graph models still vastly outperforms diffusion (+45 pt on Cora, +20 pt on PubMed; see Table 2).

Guided by the reviewer’s feedback, we now present Lighter Wave as the practical option: it preserves our physical motivation while keeping compute overhead within a few percent. We thank the reviewer for inspiring this simplification.

---

4. Responses to the reviewer’s questions

---

5 Planned manuscript updates

• Experiments – add comparison with CaIT-XXS and the results of graph benchmarks including the trends of cosine similarity for each case of Cora/Citeseer/Pubmed.
• Cost analysis – insert full table for computational cost with the lighter wave variant.
• Method - add formalization of the lighter wave update.

• All numerical details and training scripts will be released upon acceptance.

---

Remark — Ongoing fine-tuning experiments (GPT-2 / GPT-2-Large)

We are currently fine-tuning GPT-2 (124 M) and may also fine-tune GPT-2-Large (774 M) on the OpenWebText corpus, comparing the diffusive baseline with our wave update. Experiments are ongoing; full results will be reported in the camera-ready.

---

We hope these additions fully address your questions and demonstrate that the proposed wave dynamics are both scalable and efficient.

Thank you again for your encouraging review.

We thank the reviewer for the constructive comments.

We (i) clarify how our work differs from prior work, (ii) add graph benchmarks, and (iii) provide a supplemental ablation isolating FFN for velocity.

---

1. Main concerns

1. Positioning vs. prior work
2. Lack of traditional graph experiments

---

2. Relation to prior work

Table 1 Relation to prior work

• Already cited in the manuscript.
  • Graph-CON and PDE-GCN are already cited as refs [30] & [13], respectively. They investigate incorporating non-diffusive dynamics into sparse-graph GNNs, whereas our work targets complete-graph attention.
• Scientific focus.
  • Gravina et al. (2025) study over‑squashing on sparse graphs; our focus is over‑smoothing on complete‑graph. However, since it is related to our work and should be cited, we will cite it in the camera-ready version.
  • Deng et al. (2025) primarily enforce Hamiltonian constraints in Transformer-based networks for physical reasoning, and does not analyze the attention-driven dynamics of hidden states, that is the main focus of our work.
• Novelty.
  • To our knowledge, this is the first work to introduce a wave update into a Transformer with complete-graph attention.
  • Interpretations of Transformer dynamics from a continuum-physics perspective (e.g., diffusion or wave dynamics) remain scarce.
  • We further explore Velocity-FFN and Velocity-LN as optional components, completing a fully “wavy” block that extends the standard Transformer architecture.

---

3. Citation-graph benchmarks

We take DIFFormer (Wu et al. 2023), fix τ = 0.2, and add our wave update. All other hyperparameters and training scripts stay identical, isolating the effect of the wave term.

We evaluate on the three citation networks most commonly used in prior work, reporting mean ± SD over 5 seeds.

Table 2 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)

† Paired t-test (5 matched seeds): p > 0.05 → no statistically significant degradation.

▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.

τ-robustness — cases where the diffusion baseline collapses

Table 3 Graph benchmarks (DIFFormer + Wave, τ = 0.5, 5 seeds)

† Collapse: the diffusive baseline fails.

‡ 20-layer Cora and 10-layer Citeseer collapse for both models.

Key points

• When τ is large (0.5), the diffusive baseline collapses (accuracy ≈ 29 pt) on deep stacks.
• The wave update mitigates this collapse, recovering +20–31 pt (Pubmed/Cora 12-layer).
• Higher variance (Cora 12-layer/Pubmed 12-layer) arises from seed-to-seed variance during collapse recovery.

Layer-wise cosine similarity on Cora, 20 layers

Table 4 Cosine similarity (Cora, 20 layers, τ = 0.2)

As depth increases, Diffusion saturates to ≈ 1.00, showing severe over-smoothing, whereas the Wave-enhanced model stays below 0.38, preserving representational diversity and enabling the large accuracy gains highlighted above.

---

4. Ablation on ImageNet-1K (DeiT-Tiny, 150 epochs)

Table 5 Velocity-FFN ablation

---

5. Planned manuscript updates

• Related‑work: Gravina et al. (2025) will be cited in Introduction. Also, added 2×2 matrix to clarify the difference from the previous studies including papers you mentioned.
• Experiment: We will add the results of graph benchmarks including the trends of cosine similarity for each case of Cora/Citeseer/Pubmed. Also, we will added results of ablation study as an Appendix.
• All numerical details and training scripts will be released upon acceptance.

---

We hope these additions address your concerns and would appreciate it if you could kindly reconsider your rating.

Thank you again for your time and consideration.

Planned integration of the missing related work

Thank you for pointing this out. After revisiting all four papers you mentioned, we have now cited each of them. In particular, Gravina et al. and Deng et al. have been newly added (PDE-GCN & Graph-CON were already Refs [13] & [30]).

Relation to Wavy Transformer

Gravina et al. is included in Introduction as related work to our study. We also cite Deng et al. in Section 4 (Related Works)—together with the original Hamiltonian Neural Networks paper (Greydanus et al., 2019), so that readers can clearly distinguish our wave‐based approach from Hamiltonian‐constraint methods that, while also physics-inspired, follow a different conceptual path.

Large-scale graph benchmark results (3 seeds)

OGBN-Arxiv (169 K nodes · 1.17 M edges)

OGBN-Proteins (133 K nodes · 39.6 M edges)

Key observations

• Depth advantage: Wave’s gain widens as layers deepen—small loss at 3-layer Arxiv flips to +5.34 pt at 5 layers and a large +42 pt rescue at 7 layers; Proteins shows a similar +10.7 pt jump at 5 layers.
• Stability: Diffusive accuracy collapses when depth grows (68 → 62 → 24 %), whereas Wave remains stable.
• Support for our claim: Results reinforce that the wave residual mitigates depth-induced over-smoothing and is a strong practical option even for large-scale sparse graphs.

---

Clarifying scope

As noted in Section 1, lines 28–35, our goal is to interpret Transformer attention and its over-smoothing through physical intuition and, guided by that insight, introduce a wave-based alternative: the Wavy Transformer. Consequently, vision and language benchmarks (ImageNet / GLUE / BERT) remain the core of the paper, because they evaluate the effect of dynamics inside complete-graph attention mechanism.

Graph datasets serve a different purpose:

• Generality check. We use citation graphs (Cora etc.) and now OGB benchmarks (Arxiv, Proteins) to confirm that Wavy Transformer is effective on sparse graph datasets as well.
• Not the main yard-stick. We do not claim state-of-the-art on OGB; rather, the positive trend at deeper settings (+42 pt at 7-layer on Arxiv, +10.7 pt at 5-layer on Proteins) supports the wave based approach we propose.

We hope these additions resolve the remaining concern and further clarify our contribution.

We thank the reviewer for the detailed and constructive feedback.
We (i) present a full cost analysis, (ii) link reduced over-smoothing to higher accuracy, and (iii) answer your specific questions.

---

1. Main concerns

1. Computational cost – — vision-inference drops ≈ 50 %; missing NLP/train numbers
2. Over-smoothing – cosine similarity still high in Figs. 2 & 5
3. Clarifications – convergence speed, DeiT oscillation, decoder-only use

---

2. Computational cost (4 × V100) — Light vs. Full Wave

Table 1 Comparison of runtime and memory costs of BERTs

Table 2 Comparison of runtime and memory costs of DeiT-Tiny

Notes:

• Full Wave (original proposal).
  Implements a discrete Hamiltonian form of the wave equation by introducing an explicit velocity state plus Velocity-FFN and Velocity-LN. While it raises accuracy on some tasks, the extra state and auxiliary blocks slow both inference and training, as shown above.

• Lighter Wave (second-order, momentum-only).
  Responding to the reviewer’s cost concern, we removed the auxiliary Velocity-FFN/LN and kept only the momentum term
  $x_{l+1}=x_l+\operatorname{attn}(x_l)+\beta\bigl(x_l-x_{l-1}\bigr),$ where $\beta$ is the mixing parameter. This preserves the physical insight, yet matches baseline throughput (NLP -0.30 %, CV +0.5 %) and memory (NLP +2.1 %, CV +10.7 %, still < 1 GB).

• In the citation-graph benchmarks (Table 4), we employ this lighter version of the Wavy Transformer, which outperforms its diffusive counterpart in deeper settings.

Guided by the Reviewer’s cost concern, we now present Lighter Wave as the practical option: it preserves the physical insight while keeping compute within a few percent of the baseline. We thank the reviewer for inspiring this simplification.

---

3. Convergence (BERT MLM on Wikipedia + BookCorpus)

Table 3 Comparison of convergence trends

*Mix = diffusion + wave outputs; reaches 44 % 20 k steps earlier.

---

4. Citation-graph benchmarks and cosine similarity trend

We adopt DIFFormer (Wu et al. 2023) and only add our wave update (τ = 0.2). All hyper-parameters remain unchanged. We report mean ± SD over 5 seeds.

Table 4 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)

† Welch t-test p > 0.05 → no statistically significant degradation.

▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.

Full results (τ = 0.2 / 0.5) will be included in the final version.

Layer-wise cosine similarity on Cora, 20 layers

Table 5 Cosine similarity (Cora, 20 layers)

Diffusion saturates to ≈ 1.00, showing severe over-smoothing, whereas the Wave-enhanced model stays below 0.38, preserving representational diversity and enabling the large accuracy gains highlighted above.

5. Spectral gap and node-feature variance (DeiT-Tiny, ImageNet-1K val dataset: 50k images)

Table 6 Spectral gap & Node-feature variance (DeiT-Tiny, ImageNet val dataset)

Here the spectral gap is defined as $1-\lambda_{2}$, where $\lambda_{2}$ the second-largest eigenvalue (in magnitude) of the right-stochastic attention matrix $A$. A smaller gap, together with higher node-feature variance, indicates reduced over-smoothing beyond what cosine similarity alone captures.

---

6. Image classification: CaIT-XXS-24 on ImageNet-1K

Table 7 Comparison with CaIT-XXS-24

Same training recipe as CaIT; LayerScale 1e-5, no extra class-attention block.

Implementation note. By omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) yet achieves a +1 pt Top-1 accuracy gain.

---

7. Responses to the reviewer’s questions

---

8. Planned manuscript updates

• Cost and convergence analysis – insert full table for computational cost with the lighter wave variant and convergence curves.
• Experiments – incorporate graph benchmark results, layer-wise cosine similarity trends with the results of spectral gaps and node-feature variance, experimental results for the lighter wave variant and short discussion of decoder-only applicability.
• Method - add formalization of the lighter wave variant.
• All numerical details and training scripts will be released upon acceptance.

---

We hope these additions address your concerns and kindly ask you to reconsider your rating.

Thank you again for your thorough evaluation.

Thank you very much for the follow-up and for suggesting a broader evaluation of the Light Wave variant. We fully agree that showing its behavior across the full benchmark suite will make the paper stronger, and we are already running the missing experiments.

---

Ongoing runs (expected to finish before the discussion deadline)

† Diffusive is our original baseline (DeiT-Tiny); we will report its final numbers after the current Light/Full runs finish, to ensure a fair end-of-training comparison.

Interim observations

• Light Wave already matches Full Wave on ImageNet (0.04 pt lower) while keeping baseline-level speed.
• On BERT, Light Wave is +0.43 pt better than Diffusion at 60 k steps and continues to close the gap to Full Wave as training proceeds.

We are also measuring Spectral Gap $(1-\lambda_2)$ and Node-feature variance for these DeiT-tiny checkpoints (90-epoch checkpoint, DeiT-Tiny; values will be updated once training finishes).

Light Wave, though slightly less intensive, still delivers most of the over-smoothing relief at virtually no runtime cost, confirming it as a strong practical option.

Both jobs will finish before the discussion period closes. We will post the final numbers as soon as they are available.

---

Camera-ready plan (if accepted)

1. Comprehensive tables for Light Wave variant on all original benchmarks (BERT MLM, GLUE, DeiT-Tiny).
2. A concise Light vs. Full comparison of key dynamics (Cosine Similarity, Spectral Gap, Node-wise Variance) to highlight which properties are preserved.

We appreciate your suggestion and will make sure the final version includes these expanded results.

1 Full-scale benchmarks

Take-away: Light Wave is +0.9 pt ahead of diffuse baseline on ImageNet, while Full Wave retains a +1.0 pt edge in BERT MLM accuracy.

---

2 GLUE suite (3 seeds, tasks in columns)

We discovered two independent issues during final validation:

1. STS-B evaluator had loaded an earlier fine-tuning checkpoint instead of the best checkpoint.
2. Our macro averages were divided by 9 instead of 10: MNLI-m and MNLI-mm must be counted as two metrics.

After fixing both, only the STS-B row and the two macro averages changed; all other scores remain identical. The corrected results are below.

(✔ = value changed after checkpoint fix)

Observations

• Light Wave now tops both macro averages.
• Full Wave slightly outperforms Light on two tasks (SST-2, MRPC), but Light wins on seven, yielding the highest overall mean.

---

3 Behavioral metrics (DeiT-Tiny @ 300 ep)

Over-smoothing relief: Light Wave keeps a middle-range mixing rate (gap 0.73) and the lowest node variance (2.11).
Class separation: At the same time, it exhibits the largest inter-class variance (0.308 vs 0.212 / 0.195), where the variance is computed from the mean feature vector of each class. Together, the numbers imply—at least on this dataset—that Light Wave compresses local fluctuations while leaving class centres well separated, which is consistent with its accuracy lead.

---

Light Wave vs Full Wave at a glance

• ImageNet: Light +0.76 pt over Full
• BERT MLM: Full +1.03 pt over Light
• GLUE Macro Avg (10 metrics): Light +3.68 pt over Full — still +0.16 pt if STS-B is excluded
• Over-smoothing / class separation:
  Full Wave introduces stronger fluctuations that curb over-smoothing (smallest gap 0.63) yet still trails Light in inter-class variance (0.211 < 0.308).
  Light Wave achieves a middle-range gap (0.73), keeps node variance lowest (2.11), and attains the widest class-centroid spread, leading to clearer boundaries in this experiment.

---

Key take-aways

• Accuracy: Light Wave tops ImageNet and both corrected GLUE macro averages.

• Efficiency: Same level computational cost as diffusive baseline.

• The physical core of our approach remains fully consistent, and the key conclusion—that introducing wave residual mitigates over-smoothing and improves accuracy—holds unchanged.

• Practicality: Light Wave delivers the best trade-off—baseline cost, balanced smoothing, and top accuracy—making it a strong practical option.

Thank you again for the constructive feedback!

---

(Camera-ready plan unchanged: complete tables, Light vs Full comparison, public code & checkpoints.)