Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment

Dear Reviewer Rymo, thank you for your insightful questions. Below, I will respond to your concerns.

Q1: Hard-Negative Mining vs. BACL

We believe that Hard-Negative Mining is indeed conceptually related to BACL in some way, but they are distinct methods:

• Static vs. Dynamic: Traditional hard-negative mining (e.g., VSE++) selects once at top-K similarity and keeps the mix fixed; BNS re-samples every mini-batch with a learnable, differentiable curriculum α(η)→0, which provably contracts the margin (Thm 1) and stabilises early training.
• Non-differentiable vs. differentiable: Prior mining is heuristic / non-differentiable; BNS uses Gumbel-Softmax so the sampling policy is trained end-to-end.
• Global vs. Token-level: Existing miners optimise only global contrast. BACL pairs BNS with CLA, forcing token-level localisation, which classic miners cannot provide.

We conducted a simple controlled experiment (LAION-400M, ViT-B/32 backbone, 5 epochs), and the results are shown below. Even when the baseline is fed exactly the same number of “hard” samples, BACL is +8.4 pts R@1 and eliminates optimisation instability—confirming that curriculum + differentiable policy + CLA, not mere exposure to hard pairs, drives the gains.

*Loss spike = training loss surge > 4× median within 500 steps.

Q2: Training Budget & Fairness

We present the training budget as follows:

*Mixed-ratio baseline is forced to ingest exactly the final ambiguous-vs-easy distribution that BACL reaches after epoch 5, but in random order.

Controlled Result (LAION-400M, ViT-B/32 backbone)

It can be found that:

• Same epochs, same sample count: BACL’s curriculum re-orders existing data, it does not extend training length or add extra pairs.
• Injecting the same quantity of ambiguous negatives without curriculum yields +4.6 pts R@1 but still lags BACL by -6.7 pts—showing that sequence (easy→hard) + CLA, not volume, drives the improvement.
• GPU cost overhead ≤ 5 %. In practice, the extra compute holdup is negligible.

Take-away—— Therefore, BACL’s benefit is orthogonal to training budget and cannot be matched by simply “showing the baseline the same hard samples”.

Q3: Needs a pretrained model

The issue you noticed is indeed one that we have paid attention to before. However, based on our current experience, this is no longer an issue.

The coarse encoder is only used once to retrieve candidate negatives for epoch 0; afterwards, BNS and the main model co-evolve—the index is refreshed every epoch with the current weights. Our experience is that any lightweight checkpoint (or even a random init) suffices to start; the curriculum rapidly compensates.

New Ablation: Random-Init Start (LAION-400M, ViT-B/32, 5 epochs)

Even from random noise the model reaches essentially the same final R@1 (≤0.2 pt gap) after 5 epochs; the pretrained index just speeds up the first epoch’s loss but sets no ceiling.

Take-away—— BACL’s performance is not bounded by the quality of the bootstrap encoder; it merely needs an initial anchor list, which the curriculum quickly refines.

Q4: CLA applicability to pure dual-encoders

• Token-pair signal without shared attention: In a dual encoder we feed CLA the outer product of per-token saliencies $|\nabla_{h_t}s(x,y)|$ from each modality—a gradient-based substitute for cross-attention that gives the same fine-grained mismatch cue.

  • Let the similarity score be $s(x,y)=\langle \phi_X(x),\phi_Y(y)\rangle$.
    • For each input token $t$ we take the gradient \nabla_{h_t} s(x,y)$, where $h_t is that token’s hidden embedding.
    • The magnitude $|\nabla_{h_t} s|$ is a first-order surrogate of that token’s contribution (cf. “gradient-based class activation”,).
    • We compute Δ-saliency between a positive pair and its hardest negative .

• Why this is principled:

  1. Gradients are directional derivatives of similarity; they highlight the minimal perturbation that would flip a match to a non-match, i.e., the same boundary concept CLA targets.
  2. The operation is fully differentiable and incurs <1 % runtime overhead (one backward pass per anchor).
  3. It preserves CLA’s loss unaltered—only the source of the token weights changes.

Take-away—— CLA is architecture-agnostic: in decoupled encoders it simply swaps “cross-attention gaps” for “similarity-gradient gaps”, retaining the fine-grained supervision without imposing a shared attention module.

Thank you again for your valuable comments and your precious time. We hope our clarification has addressed your concerns.

Dear Reviewer Rymo,

Thank you once more for your insightful feedback. We hope the clarifications in our rebuttal fully addressed your concerns. If any questions remain or further discussion would be helpful, we would be glad to elaborate. We hope these clarifications assist in your re-evaluation.

Best regards

Dear reviewer o7kP, thank you for your valuable comments and recognition of our work. Below, we will respond to your concerns.

Q1 : CLA Efficiency

We appreciate the reviewer’s concern about potential slow-downs. CLA re-uses the final‐layer token embeddings already produced by the two uni-modal encoders. A single einsum (≈ O(d·n²)) generates the cross-attention heat-map; its cost is on par with the in-batch cosine-similarity that CLIP already computes.

Measured impact (LAION-400M, A100-40 GB, batch = 512):

End-to-end epoch time rises by < 8 %, which is well below the “much slower” threshold implied in the comment. Therefore, we believe that CLA achieves a good balance between performance and efficiency.

Q2: BNS Runtime & Complexity

Measured impact (LAION-400M, 8 × A100-40 GB, batch = 512):

As can be seen from the table above, the overhead is minor, and the explanation is as follows:

• CPU-side, vectorised top-k only. BNS ranks in-batch negatives with a single torch.topk on 16-bit logits; per-step wall-time ≈ 0.4 ms.
• No extra GPU forward/backward. The policy network is a 2-layer MLP (0.9 M params ≈ 0.05 % of CLIP-B). Gradients flow through the existing embeddings; memory and FLOPs are negligible relative to the vision backbone.

Hence BNS increases epoch time by < 4 %—well within typical variance for distributed training.

Q3: GPU Memory

Peak memory on a single A100-40 GB (batch = 512):

Full BACL raises peak GPU memory only modestly: on a 40 GB A100 we observe 29.6 GB for the baseline, 30.0 GB with BNS, 31.1 GB with CLA, and 31.3 GB for the full model—an increase of < 1.7 GB that reduces the maximum batch from 512 to 480 (-6.3 %). The extra footprint stays small because CLA builds its cross-modal map on half-resolution patch tokens and applies gradient checkpointing, while BNS is a 0.9 M-parameter MLP that resides on the CPU and contributes negligible GPU tensors. If practitioners need the original batch size, commonplace techniques such as flash-attention, micro-batching, or pure FP16 reclaim the memory headroom without code changes.

Thus, BACL’s memory overhead is modest and easily managed in typical training setups.

Q4: 0-shot retrieval on ImageNet, COCO

We have followed exactly the evaluation protocol and metrics of M4(ICLR ’24). As can be seen from the results in the table below, under the identical zero-shot setup, BACL surpasses the SOTA.

In fact, ImageNet and COCO/Flickr contain far fewer ambiguous negatives than noisy web data (which is mostly the case for the data used in our original experiments). Through this experiment, we also found that BACL’s margin-shrinking objective universally improves representation quality—not only when hard negatives are abundant, but also when the evaluation data are "clean".

Thank you again for your valuable comments and your precious time.

Dear reviewer cghr, we greatly appreciate your recognition of our work and its strengths. Now, we would like to respond to your concerns.

Q1: Interpretable Evidence for CLA

Your concern is very meaningful. Due to the rebuttal policy, we cannot provide images in any form, so we will offer some textual explanations here.

For each modality (image↔text, video↔text, audio↔text) the next version appendix will add a grid with (i) positive-pair attention, (ii) hardest-negative attention, and (iii) the ΔA map with the top-10 cells boxed. The mismatching caption tokens / frames are printed alongside, letting readers verify that CLA focuses precisely on the erroneous region.

From the checkpoints used in Table 1 we compute Alignment-Error-Localization (AEL)—percentage of human-tagged mismatch tokens covered by the top-10 % ΔA cells.

These results demonstrate that CLA localises fine-grained mismatches far more accurately than the baseline.

Q2: Dataset scale

As far as we know, our core benchmarks are already web-scale; at the very least, we have adopted the same configurations as some existing SOTAs.

• LAION-400M – 400 million image–text pairs (same corpus used by CLIP/ALIGN).
• WebVid-10M – 10 million video clips.
• VAST-27M – 27 million tri-modal samples.
These sizes equal or exceed those in recent SOTA works (GRAM, Emergence, M3-JEPA).

We additionally train CLIP ± BACL on three LAION subsets (identical hyper-params, 5 epochs).

The ~30 % improvement is stable from $10^8$ to $10^9$ samples, showing BACL is not a small-data artefact.

Q3: Truly ambiguous

For every anchor x we compute a boundary score
 BS(x,z)=sim(x,z)−sim(x,y⁺).
If an alternative caption z is as close as the ground-truth (BS ≤ 0) then, once the curriculum coefficient α(η)<0, its adjusted weight û becomes ≤ baseline and the Gumbel-Softmax assigns ≈0 probability. These “equally-valid” captions are therefore never punished.

We sample 1 000 LAION-400M images that each have ≥5 near-duplicate captions (human verified as co-valid).
Metric = False-negative rate = fraction of co-valid captions ranked below top-k by the model (k=5).

We can find that BACL reduces erroneous rejection of multi-ground-truth captions by 0.6 pp.

In addition, some evidence is provided in the theoretical section of the paper. Assumption 1 explicitly allows ρ→1 (all negatives ambiguous). The proof of Theorem 1 shows the fast-rate bound holds because samples with BS≤0 remain inside the decision margin—they are never forced across it. Thus BACL is provably safe for inherently ambiguous data.

Q4: Future work discussion

Very interesting discussion. Here we will present our ideas.

In the instruction-tuning scenario we can simply interpret the answer as the second “modality”.
A training triple now consists of an instruction I, its reference answer A* (the positive), and a near-miss answer A⁻ (the negative).

Negative sampling. For each (I, A*) we retrieve semantically close but not identical answers from an embedding index (or generate them with the same language model).
We compute a boundary score
  sim(I, A⁻) – sim(I, A*).
The BNS uses this score with the same logistic schedule as in the multimodal case: early epochs focus on easy negatives, later epochs concentrate on harder near-misses.

Fine-grained disambiguation. Because the model is a decoder-style LM, we already have cross-attention maps from instruction tokens to answer tokens. CLA compares the attention pattern of (I, A*) with that of (I, A⁻); the ΔA matrix highlights exactly the answer spans where the near-miss contradicts the reference. The local mismatch loss then directs updates to those tokens, encouraging the model to correct the specific error rather than rewriting the whole answer.

Safety for valid paraphrases. If an alternative answer is truly equivalent to the reference, its similarity is at least as high, making the boundary score non-positive. Once the curriculum coefficient α(η) becomes negative, such cases receive zero weight and are never penalised.