DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

Response to Reviewer $\color{green}\text{gFQs}$:

We sincerely thank you for your invaluable and constructive feedback. We particularly appreciate your positive acknowledgement of our novelty, clear organization, and extensive experimental validations. Below we provide the point-to-point responses to address your concerns about our approach.

In-depth Analysis of $\beta$. To deepen the understanding of the proposed approach, the authors could provide more detailed analysis to demonstrate how the data- and model-aware strategy influences performance...

Thank you for pointing this out. Your suggestion is invaluable for this paper. To address this, we show the dynamic $\beta$ concerning the data hardness, model responses, and the combination in the Figures 4,5,6 of the Anonymous Link.

• 1. $\beta_{D}$ for the data hardness (Figure 4 in Anonymous Link). From Figure 4, we can observe that the range of $\beta_{D}$ falls within (0.0524, 0.1428) with the original $\beta$ initialized as 0.1. Moreover, we observe that the mean value is 0.0999 with a standard deviation of 0.0288. These observations demonstrate that $\beta_{D}$ maintains proximity to the original $\beta$, while adaptively adjusting based on the data characteristics, enabling a more effective capture of the data.

• 2. $\beta_{M}$ for the model responses (Figure 5 in Anonymous Link). From Figure 5, we can find that as the training progresses, the $\beta_{M}$ gradually converges to the original $\beta$. Meanwhile, we find that the $\beta_{M}$fluctuates within a moderate range of (0.0530, 0.1580), demonstrating controlled adaptivity. These observations suggest that as the model training stabilizes, its responsiveness becomes more consistent and eventually approaches a steady state.

• 3. $\beta_{C}$ by combining both (Figure 6 in Anonymous Link). From Figure 6, we can observe that combining both the data- and model-aware strategies yields a more dynamic range of beta values, spanning from 0.0177 to 0.2261, which is wider than either. Moreover, while the value eventually stabilizes around 0.1, we notice that the mean value during the training stage is a bit lower than $\beta_{M}$. This suggests that it relaxes the constraints during training based on the data hardness, enabling the model to better capture fine-grained data patterns and thereby adaptively enhancing its responsiveness to data characteristics.

Thank you again for your kind suggestions and supportive feedback. If you have any additional questions, we would be pleased to discuss them with you.

Response to Reviewer $\color{red}\text{TUJo}$:

We highly appreciate your insightful comments and acknowledgment of our contributions! Your constructive criticism is invaluable in refining our work! We organize your concerns into the following 3 aspects:

Q1. Clarification about sub-sentence construction.

Thank you for pointing this out! We provide our detailed prompt template as:

You are an expert in extracting facts from the given question-answer pair about an image. Your task is to: Analyze the provided question-answer pair based on the image, extract all factual statements from the answer, and rewrite them into self-contained sentences.
\n\n Requirements for each sentence are: \n1. complete, each sentence must be self-contained; \n2. factual (omit opinions, subjective statements); \n3. concise (no more than 77 tokens).
\n\n Format your result strictly as:\n### Facts:\n- {Fact 1 (e.g., "A red shoe sits on a wooden floor.")}\n- {Fact 2 (e.g., "The shoe has laces and a white sole.")}\n- ...\n\n### Question-answer pair: Question: "{Question}" Answer: "{Answer}"

For the very limited number of sub-sentences with more than 77 tokens (2/113156 for preferred, 1/114268 for rejected), we apply truncation over them. Here is a representative example demonstrating our sub-sentence construction:

"question": "Is this book related to Literature & Fiction?",
"answer": "No, this book is not related to Literature & Fiction. It is a religious or theological book, as evident from the title \"What Love Is This? Calvinism's Misrepresentation of God\" by Dave Hunt."
"facts": [
"The book is not related to Literature & Fiction.",
"The book is a religious or theological book.",
"The title of the book is \"What Love Is This? Calvinism's Misrepresentation of God\" by Dave Hunt."]

Q2. Evaluation of different sub-sentence granularities.

Thank you for pointing this out! To validate this, we modify the prompt template by replacing the 77 in no more than 77 tokens with 60 and 50.

Table 1: Token Length of the segmented sub-sentences from the preferred responses (22626 responses in total)

Table 2: Token Length of the segmented sub-sentences from the rejected responses (22626 responses in total)

Tables 1 and 2 show the statistics of the segmented sub-sentences. The statistics strongly support the robustness of our approach, with over 99% of sub-sentences containing fewer than 40 tokens, and more than 63% having fewer than 20 tokens.

Due to the inherent inability of the LLM to precisely handle the length constraints [1], we manually truncate the sub-sentences exceeding the length constraints at different scales. The distributions of $\delta$ are in Figures 1,2,3 of the Anonymous link, and subtle differences in $\delta$ can be observed over different segmentation strategies.

Table 3: Performance over object-hall bench.

Moreover, we also evaluate the model trained with the given different $\delta$ in Table 3, and observe subtle differences. These comprehensive statistics and experimental results demonstrate DAMO's robustness across different segmentation granularities.

Q3. Typo Correction. Line 208 (right): "the momentum $\gamma$ is set to 0.9, and $\bar{H}$ is initialized to 0." $\bar{H}$ should be $\bar{R}$.

We acknowledge this typographical error and have corrected this in the revised paper.

Thank you again for your valuable and insightful suggestions. We welcome any additional questions and would be happy to discuss them further.

[1] Yuan et al. "Following length constraints in instructions."

Response to Reviewer $\color{blue}\text{ERND}$:

We highly appreciate your insightful comments, which help us a lot to better scrutiny and polish our work! The following are point-to-point responses.

Q1. Implementation with more advanced models (e.g., LLaVA 1.6 and LLaVA-OneVision) makes DAMO more compelling.

Thank you for your kind suggestion. We extend DAMO to more advanced models like LLaVA 1.6 and LLaVA-one-vision with a more comprehensive alignment dataset[1]. Due to the GPU constraint, only the MLP layer is finetuned for LLaVA-one-vision, and all models are finetuned with one epoch. As shown in Table 1, DAMO still demonstrates consistent performance gains across these advanced models.

Table 1: Performance on the Object-hal bench.

Q2. Measurements over more helpfulness benchmarks further support our effectiveness.

Thank you for your kind suggestion regarding model helpfulness. Our evaluation on LLaVA-Bench and MM-Vet (Table 4 of the paper) shows that both our 7B and 13B models achieve competitive performance, with notable improvements of 6% and 9% over their baseline counterparts, respectively.

Furthermore, we extend the helpfulness evaluation to the MME Bench with advanced models. As shown in Table 2, DAMO consistently improves both perception and cognition capabilities across different model architectures, demonstrating its effectiveness in enhancing both helpfulness and hallucination resistance.

Table 2: Performance on the MME bench.

Notes: We sincerely appreciate your valuable feedback regarding the helpfulness and advanced models. Our extensive experiments demonstrate that DAMO achieves consistent and significant improvements across:

1. Multiple model architectures (LLaVA-1.5 / 1.6, LLaVA-one-vision)
2. Extensive benchmark suites, including:
   • Helpfulness (MME, MMVet, LLaVA-Bench)
   • Hallucination (Object-Hal, AMBER, MM-Hal)

Most importantly, DAMO serves as a plug-and-play mechanism that can be seamlessly integrated into various architectures with different alignment data, while maintaining minimal computational overhead. This versatility and efficiency, combined with consistent performance gains across different settings, support the effectiveness and practical value of DAMO.

Q3. Adding essential related work. Understanding Alignment in Multimodal LLMs: A Comprehensive Study, Amirloo et al., 2024.

Thank you for providing this related work. We have added this method to our revised paper, As a representative method, BDHS significantly advances the MLLM community by pioneering the alignment techniques into the advanced LLaVA 1.6 architecture, effectively bridging the gap between theoretical MLLM alignment research and practical applications.

Q4. Clarification about the model-aware strategy.

Sorry about the confusion. Let us clarify our model-aware strategy that adjusts $\beta$ according to the implicit reward gap between the preferred $y_w$ and rejected $y_l$. Specifically: 1. A larger $\beta$ is assigned to a larger reward gap between $y_w$ and $y_l$, which indicates that the model has already grasped this type of response well. 2. A smaller $\beta$ is assigned to a smaller reward gap between $y_w$ and $y_l$, which suggests that the model needs to improve its responsiveness on such cases. This adaptive scaling mechanism helps the model focus more on cases with less confidence, while maintaining its performance on well-learned cases.

Q5. Presentation refinement.

• Please do not mix serif (body-text, Figure 4) and non-serif font (Figure 1, 2, 3).

Thank you for pointing this out. We have unified the font in our revised paper.

• L081: "Similarly to Section 3.1". It is odd to see this statement in Section 1.

Thank you for your kind suggestions. We have removed such descriptions and polished this as discussed in Q4.

• All Section 2 could be removed ...

While this section provides essential background for researchers new to preference alignment in MLLMs, we have significantly condensed it to improve paper conciseness.

• Equation 8, is there an extra dot after D?

Sorry about the confusion. The first dot is the dot product, and the second dot is the period at the sentence end. We have modified as $\beta_{D} = \beta \times \alpha_{D}$.

• one could appreciate to have a panel of responses in the appendix ...

Thank you for constructive suggestions, we have added these analyses to the revised paper.

[1] Yi-Fan Zhang etal MM-RLHF: The Next Step Forward in Multimodal LLM Alignment, 2025.