Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models

Dear Reviewer 5PqY,

We sincerely appreciate your constructive feedback. We are pleased that you found our method to yield robust and consistent improvements across benchmarks and LLMs. We would like to address your points in detail below.

[Q1] Ablation studies in Table 6 lack a fundamental experiment to isolate the role of the Image-to-Response Guided Loss.

[A1] To isolate the effect of the Image-to-Response Guided Loss, we conducted an additional experiment by training LLaVA-1.5 with only the next-token prediction loss and the Image-to-Response Guided Loss, excluding our proposed shuffle learning tasks. This setting allows us to directly assess the contribution of the guided loss to visual-textual alignment. The resulting performance on ScienceQA-IMG and TextVQA shows improvement over the LLaVA-1.5 baseline, but remains lower than our full Direct-LLaVA model. This result suggests that while the loss improves visual grounding, the Image-to-Response loss alone may not be sufficient for robust multimodal reasoning.

[Q2] Computational cost: Direct-LLaVA trained with the same number of GPUs as LLaVA 1.5.

[A2] We clarify that our experiments were conducted using A100 GPUs with 40GB of VRAM, not 80GB. We employed 32 GPUs primarily to accelerate training and support the breadth of ablation studies, not to increase batch size or model capacity. Importantly, despite the additional compute, we strictly maintained the same training hyperparameters as LLaVA-1.5, including batch size, learning rate, optimizer, and number of epochs, to ensure a fair and controlled comparison.

[Q3] The use of purely random permutations

[A3] In the early development phase, we have already experimented with purely random permutations sampled from the full permutation space. However, the model failed to converge under this setting. This is primarily because visual random permutations are extremely hard to predict, especially when the image patches are rearranged into highly disordered configurations with no spatial continuity. This randomness creates an enormous and unstructured search space for the permutation prediction task, resulting in unstable and ineffective learning objectives. Moreover, many of these permutations are either too trivial or too chaotic, providing inconsistent or uninformative supervision. To address this, we adopted a more structured strategy by sampling 10,000 permutations with controlled Hamming distances, following established practices in self-supervised learning (e.g., [1, 2, 3]). This design ensures a more learnable yet sufficiently diverse permutation space, enabling better training stability and performance. To validate this choice, we conducted an ablation varying the number of permutations. As shown below, increasing from 5,000 to 10,000 leads to improved performance on ScienceQA-IMG and TextVQA, confirming that a richer permutation space helps the model better learn spatial relationships. However, increasing the number to 15,000 leads to only slight improvements, indicating that 10,000 well-chosen permutations already provide sufficient diversity for effective learning, and additional permutations may have limited impact.

[Q4] Presentation of some tables and figures

[A4] We will revise the figures and tables to ensure that all numerical values are clearly legible in the camera-ready version. Regarding Table 4, its purpose is to highlight the impact of different large language models (LLMs) under a unified setting (Direct-LLaVA and LLaVA baselines) to isolate the effect of the LLM backbone. While some values overlap with Tables 2 and 3, Table 4 provides a focused comparison across models using the same architecture and training setup, which is not the emphasis of Tables 2 and 3.

[Q5] It would be interesting to repeat the experiment from Table 1 with Direct-LLaVA too.

[A5] We conducted the same experiment as in Table 1 by replacing the visual input with a black image and evaluating Direct-LLaVA. However, unlike LLaVA, our model is explicitly trained to depend on meaningful visual input through image order reconstruction and Image-to-Response Guided Loss. As a result, when presented with blank images, Direct-LLaVA consistently refuses to generate an answer or outputs an empty response, leading to 0% accuracy across test samples. This behavior reflects the model's learned reliance on visual grounding and its sensitivity to missing or meaningless visual inputs. For this reason, we report the result as N/A to indicate that the model does not produce valid outputs under such conditions, rather than interpreting the result as typical task performance. We will clarify this behavior in the revised version.

References

[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.

[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.

[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022.

Dear Reviewer kMkc,

We sincerely appreciate your thoughtful and encouraging feedback. We are pleased that you appreciate the novelty of our modality-shuffling framework and the effectiveness of our guided loss in improving visual-text alignment and reducing language bias. Your comments are deeply valued, and we address your specific points below.

[Q1] The core idea of this paper is simple and intuitive, but the overall writing quality appears rushed and needs further polishing.

[A1] Thank you for the constructive feedback. In the revised version, we will carefully reformat all figures and tables as suggested to ensure better readability. We appreciate the reviewer’s attention to these details and will ensure a polished revision.

[Q2] The Related Work section is overly generic regarding large multimodal models.

[A2] We thank the reviewer for the valuable suggestions [4, 5, 6, 7]. In the revised version, we have incorporated a more structured discussion of alignment paradigms in LMMs, including cross-attention mechanisms (e.g., Flamingo, CogVLM), Q-former-based alignment (e.g., BLIP-2, MiniGPT-4), and MLP-based projections (e.g., LLaVA, Qwen2-VL). These additions help position our work more clearly within the broader landscape of multimodal learning.

[Q3] The image permutation reconstruction task lacks sufficient implementation detail.

[A3] We have detailed our implementation in L163–165. We denote the permutation index as $k$, which represents the specific ordering of image patches applied via a permutation function $P(x, k)$. To predict $k$, we introduce a visual order projection head, implemented as a linear layer applied to the last hidden-state representation of the directed token. The model learns to regress the permutation index $k$ from this representation.

[Q4] Ablations on the number of permutations and patch size.

[A4] Our selection of 10,000 permutations follows the well-established practice in self-supervised vision learning, particularly works on jigsaw-based tasks [1, 2, 3]. Specifically, we sample permutations to maintain a balanced Hamming distance between them, ensuring sufficient diversity while avoiding overly trivial or degenerate cases. To validate this choice, we conducted an ablation varying the number of permutations. As shown below, increasing from 5,000 to 10,000 leads to improved performance on ScienceQA-IMG and TextVQA, confirming that a richer permutation space helps the model better learn spatial relationships. However, increasing the number to 15,000 leads to only slight improvements, indicating that 10,000 well-chosen permutations already provide sufficient diversity for effective learning, and additional permutations may have limited impact.

[Q5] Two loss terms appearing simultaneously in Equation 13 introduce ambiguity.

[A5] Eqn (13) is not ambiguous. We clarify that although Eqn. (13) presents the three all loss components in a single expression for conciseness, they are not applied simultaneously within a single forward-backward pass. As clarified in Algorithm 1 (included in the revised version), each training sample is assigned to only one task per step: either autoregressive prediction, image order reconstruction, or text order reconstruction. These tasks are handled in independent forward-backward passes, and the corresponding loss is computed based on the selected task. Eqn. (13) is intended to summarize the overall training objective across the dataset, not a per-step computation. We will update the text surrounding Eqn. (13) to better reflect this and avoid potential confusion.

[Q6] Analysis of failure cases or potential trade-offs.

[A6] We conducted an additional experiment using LLaVA trained with only the Image-to-Response Loss, without incorporating our shuffle learning tasks. The resulting performance on SciQA-IMG and TextVQA is lower than that of our full Direct-LLaVA model, but still demonstrates performance improvement over the LLaVA baseline. This result suggests that while the loss improves visual grounding, the Image-to-Response loss alone may not be sufficient for robust multimodal reasoning.

[Q7] Some experimental results in Table 3 appear incomplete or too limited in scope.

[A7] We acknowledge that rows 4, 5, and 6 in Table 3 appear limited in scope due to missing metrics. These entries correspond to baseline models whose full results are not publicly disclosed in the original papers; thus, we have included the available results for completeness and reference. In the revised version, we will clarify this in the table caption and main text. We are open to removing these rows if the limited reporting causes confusion or detracts from clarity.

[Q8] Minor issues

[A8] We would like to thank the reviewer for the suggestion. We have revised and updated the minor issues to improve the presentation of our paper.

References

[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.

[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.

[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022.

[4] Wang et al. Cogvlm: Visual expert for pretrained language models. NeurIPS, 2024.

[5] Wang et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv, 2024.

[6] Li et al. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.

[7] Deng et al. Visual grounding via accumulated attention. CVPR, 2018.

Dear Reviewer Xcbu,

We sincerely appreciate your thoughtful and encouraging feedback. We are pleased that you found our proposed method a simple yet effective approach to cross-modal integration, with strong motivation and compelling empirical support. Your recognition of our experimental rigor and comprehensive benchmarking is truly appreciated. We value your comments and would like to respond to your points as follows.

[Q1] The paper's structure and formatting can be improved.

[A1] We agree that some structural improvements could enhance readability. In the revised version, we will restructure our paper as your suggestion to improve the presentation of the paper.

[Q2] The paper lacks deeper theoretical justification or analysis regarding why the Directed Token facilitates better cross-modal alignment.

[A2] We thank the reviewer for the thoughtful comment. The directed token is theoretically motivated by the causal attention constraint in autoregressive LMMs, where earlier tokens (e.g., visual inputs) cannot attend to later ones (e.g., text). By placing a learnable token at the end of the sequence, it has full access to both modalities and functions as a global integrator of cross-modal information. From an information-theoretic view, it acts as a bottleneck token, summarizing joint semantics for downstream tasks. This design aligns with our experimental findings. As shown in Table 5, using the directed token significantly improves performance over standard visual tokens.

[Q3] It would be better to have a complete algorithm box.

[A3] We appreciate your suggestion. The subsequent section elucidates the algorithm employed during the two-stage training process.

Algorithm: Training Step in the Two-Stage Training Procedure of Direct-LLaVA

Input:

• Initial model parameters: $\theta$
• Training dataset $\mathcal{D} = \{(x, p, \\{(x_q^t, x_a^t)\\}_{t=1}^T)\}$
• Number of training iterations: $N$

Output:

• Updated model parameters: $\theta^*$

---

for each iteration do

1. Sample a training instance $(x, p, \\{(x_q^t, x_a^t)\\}_{t=1}^T)$ from $\mathcal{D}$

2. Autoregressive Task with Cross-Entropy and Guided Attention Loss
   a. Construct instruction input:
       $x\_{\text{instruct}} \leftarrow \text{GetInstruction}(\text{`ARTask'})$ (Eq. 3)
       $x_a \leftarrow p$
   b. Compute losses:
       $\mathcal{L}\_{\text{CE}} \leftarrow \text{CrossEntropy}(x, x\_{\text{instruct}}, x\_a)$ (Eq. 4)
       $\mathcal{L}\_{\text{I→R}} \leftarrow \text{ImageToResponseLoss}(x, x\_a)$ (Eq. 12)
   c. Update model:
       $\theta \leftarrow \theta - \nabla\_\theta (\mathcal{L}\_{\text{CE}} + \mathcal{L}\_{\text{I→R}})$

3. Image Order Reconstruction Task
   a. Construct instruction:
       $x\_{\text{instruct}}^{\text{image}} \leftarrow \text{GetInstruction}(\text{`ImageTask'})$ (Eq. 7)
   b. Shuffle image:
       $\bar{x} \leftarrow \mathcal{P}(x, k)$
   c. Compute loss:
       $\mathcal{L}\_{\text{Image-Order}} \leftarrow \text{PredictPermutation}(\bar{x}, x\_{\text{instruct}}^{\text{image}})$ (Eq. 6)
   d. Update model:
       $\theta \leftarrow \theta - \nabla\_\theta (\mathcal{L}\_{\text{Image-Order}})$

4. Text Order Reconstruction Task (Pre-training Only)
   a. Construct instruction:
       $x\_{\text{instruct}}^{\text{text}} \leftarrow \text{GetInstruction}(\text{`TextTask'})$ (Eq. 8)
   b. Construct answer:
       $x\_a^{\text{text}} \leftarrow `concat`(p, \mathbf{drt})$
   c. Compute loss:
       $\mathcal{L}\_{\text{Text-Order}} \leftarrow \text{ReorderPrediction}(x, x\_{\text{instruct}}^{\text{text}}, x\_a^{\text{text}})$ (Eq. 9)
   d. Update model:
       $\theta \leftarrow \theta - \nabla\_\theta (\mathcal{L}\_{\text{Text-Order}})$

end for

[Q4] How is the balance among multiple losses tuned? Are these loss weights fixed or adaptively adjusted during training?

[A4] In our current implementation, we assign equal weighting to all loss components (cross-entropy, image/text order reconstruction, and attention-guided loss) for simplicity and fairness in evaluating their contributions. As noted in our limitations, we have not yet explored loss weighting strategies. Despite this, our model still achieves consistent performance gains across all benchmarks (Tables 2–3), indicating the robustness of the proposed objectives even under uniform weighting. We reserve the investigation of loss weighting for future work.

Dear Reviewer oM4U,

We sincerely appreciate your positive feedback. We are glad you appreciated our novel learning objectives and directed token design, and the robustness of our results across different benchmarks. We are also grateful that the clarity of our methodology and thorough ablation studies were particularly notable. Your comments are very much appreciated, and we would like to address your points as follows.

[Q1] Alternative placements of the directed token

[A1] We chose to place a single directed token at the end of the input sequence to ensure it attends to the full context from both visual and textual modalities under the autoregressive modeling constraint. As shown in Section 3.1, placing the directed token after all other tokens allows it to aggregate multimodal information effectively, which would not be possible if positioned earlier due to the causal attention mask. Our ablation study in Table 5 validates this design: using directed tokens yields consistent performance gains over visual tokens across all benchmarks. In addition, to further justify this design, we conducted an additional experiment comparing placements at the beginning and end of the sequence. Results show that placing the token at the end yields superior performance, supporting our choice.

[Q2] Visualization of multimodal misalignment

[A2] While visualization is restricted by the rebuttal policy, we have analyzed failure cases through systematic ablations and benchmark results. As shown in Table 1, LLaVA-v1.5 exhibits minimal performance drop when visual input is replaced with a black image, highlighting misalignment where the model overly relies on language priors. Our ablation studies in Table 6 further indicated that removing the proposed Image-to-Response Guided loss or shuffle learning leads to performance degradation across vision-intensive benchmarks (e.g., SciQA drops from 74.3% to 72.3%), indicating residual multimodal gaps. We will incorporate more visualization in our revised versions.

[Q3] The shuffle learning tasks may not fully represent semantic misalignment encountered in real-world scenarios.

[A3] While the tasks are constructed in a controlled manner, they are inspired by well-established techniques in vision learning, where solving shuffled inputs (e.g., jigsaw puzzles) has been widely used to improve spatial reasoning and feature generalization [1, 2, 3]. Our adaptation of this paradigm to multimodal settings aims to explicitly address real-world misalignment issues, as shown in Table 1, where LLaVA performs similarly with or without visual input. By forcing the model to reconstruct the correct order of image patches or textual segments, our method strengthens the visual-textual grounding. This is further supported by the performance improvement on real-world benchmarks (Table 3), indicating the effectiveness of shuffle learning beyond synthetic settings.

[Q4] The main components are individually incremental, and the novelty lies more in the combination than in fundamental algorithmic innovation.

[A4] We respectfully but strongly disagree with the reviewer’s assessment. The novelty of our work lies in the synergistic integration of shuffle learning, directed tokens, and guided attention into a unified framework that addresses the underexplored challenge of modality misalignment in LMMs. Our approach spans both image and text modalities through image order reconstruction and text order reconstruction tasks during pre-training. It continues with image order reconstruction during fine-tuning, enabling consistent cross-modal alignment across training stages. The proposed Image-to-Response Guided Loss further enforces visual grounding via attention supervision, a mechanism not explored in prior LMMs. As reflected in performance gains across 10+ real-world benchmarks (Tables 2–3), this integrated design delivers practical impact beyond incremental additions. Moreover, Reviewer 2 highlights the elegance of the directed token under causal attention; Reviewer 3 notes the effectiveness of coupling it with shuffle learning; and Reviewer 4 affirms that our combination of tasks and losses yields consistent improvements, even when model backbones vary.

References

[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.

[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.

[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022.