Q-Adapt: Adapting LMM for Visual Quality Perceiver with Progressive Instruction Tuning

1. In Table 1, Q-Adapt-3B is worse than Co-Instruct-8B with Co-Instruct dataset for training. Can you explain this phenomenon? Considering that Co-Instruct have more data than Q-Instruct. Does this mean that your method will fail (i.e., worse than Co-Instruct-8B) when the amount of data is large enough? If it is due to the amount of model parameters, is it possible to conduct a comparative experiment using the same LLM as Co-Instruct-8B?
2. Lack of ablations. From Table 7, i notice that joint training (69.73) is better that Perception training (68.89) but worse than Quality training (73.41) in Stage 1. However, there is a lack of only quality training experiments in Stage 2. So, i wonder if joint training is the best method in stage 2? What about joint training v.s. quality training in stage 2?

[1]The motivation section needs improvement. In Line 51, the conflict between the two EIQA tasks is mentioned without explaining why this conflict arises or giving examples. This should be clarified in the introduction rather than in Line 220.

[2]It’s unclear what the proposed method gains from the perception answering task in the second phase. The paper stresses the importance of the explanation task and highlights the negative impact of the perception answering task during joint training. If this is the case, why include the perception answering task at all? The authors should provide a detailed analysis of the knowledge required for both tasks, as well as how they benefit from and conflict with each other.

[3]The proposed V-T Generator and T-V Prompter are central to the paper, but their approach seems similar to [1]. While this might be new for the IQA field, the contribution to the broader CV and MLLM communities appears limited.

[4]The results in Figure 3 are confusing. The caption refers to the quality explanation task, but Line 216 and the y-axis suggest these are results for the attribute-wise perception answering task. This needs clarification.

[5]In the experiments presented in Figure 3, was the data volume kept consistent across the two tuning settings? Assuming that Figure 3 indeed represents results for the attribute-wise perception answering task, why does the model tuned specifically for this task perform the worst? Could this be attributed to differences in data volume?

[6]The reviewer suggests merging the paragraph starting at Line 244 in Section 3.3 with Section 3.4. The current organization may mislead readers into thinking that the connector is responsible for adaptively selecting the required perceptual knowledge based on task instructions

[7]How significant is the impact of using MLP versus Q-Former as the connector on the final results?

[8]Table 7 lacks an ablation study that focuses solely on fine-tuning with the quality explanation task in the second stage. This would help evaluate whether including the attribute-wise perception answering task is beneficial to model training.

[9]Based on the results reported in Table 2, the sum score for Qwen-VL-Max should be 5.18, not the current value. The reviewer recommends carefully verifying the reported data throughout the paper.

[10]Including more comparisons with other MLLM performance benchmarks would further strengthen the paper. For instance, comparisons with models such as LLaVA-OneVision [2], LLaVA-NeXT-Interleave-7B [3], and Qwen2-vl [4] would be highly beneficial.

[11]Minor Weaknesses:

• The use of "(i)" and "(ii)" in the second paragraph of the Introduction appears multiple times, which disrupts the flow and readability. The reviewer suggests varying the symbols when listing multiple points.
• There is a typo in Table 6, second row: "$Q$-$Former^{Co}$" should be "$Q$-$Former^Q$."

[1] Instruction Tuning-free Visual Token Complement for Multimodal LLMs.

[2] Llava-OneVision: Easy Visual Task Transfer.

[3] Llava-Next-Interleave: Tackling Multi-Image, Video, and 3D in Large Multimodal Models.

[4] Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution.

1. The term "progressive" is a little misleading; it is simply a two-stage training method.

2. The motivation for this work is unclear. The authors claim that the two examined tasks are conflicting, which is not well justified. From the reviewer's perspective, the two tasks could complement each other: a holistic understanding of visual quality aspects of the image can enhance more detailed quality assessment tasks involving local image analysis, and vice versa. This indeed motivates the authors to propose the two-stage training method.

3. The use of LoRA in the first stage of training is not clearly explained. Why is parameter-efficient fine-tuning (such as LoRA) necessary? Which subset of parameters is subject to LoRA fine-tuning, and how are these parameters identified?

4. The information flow in stage two (Figure 2) appears somewhat redundant. For example, both raw visual features and processed visual features (by the V-T generator) are sent to the T-V prompter, and then combined with the raw features to feed into the language model. Is such a complex design necessary? Further justification would be appreciated.

5. The generated quality-relevant textual descriptions should be evaluated for both correctness and diversity. Template-like textual outputs are unlikely to be perceived as explainable.

6. The experimental setups need clearer descriptions. For instance, how are the competing methods implemented?

7. How were the visualizations in Figure 4 generated, and how should these results be interpreted and compared?

1. The symbols in some equations are not explained clearly, such as n and m near line 196 representing the dimension of feature space.
2. The legend is completely consistent with the abscissa label in the Figure 3, so the authors are advised to remove the legend.
3. It is suggested that the authors add an experiment in Sec 3.2, using the same method as in Figure 3 to perform finetune on another task, attribute-wise perception answering task, to prove that joint finetune will affect the performance which is mentioned in the conclusion 1 given by the authors in Sec 3.2. At present, it can only be proved that adding perception negatively affects the overall quality task.
4. The left values are exactly the same for Equ.3 and Equ.7. Although the author declares "update" above Equ.7, it is also recommended to use different symbols for the left values of Equ.3 and Equ.7 to avoid confusion.
5. It seems that the authors forgot to mark the better results as bold in the top right grid of Table 6.
6. The sentence starting with “We analyze the effect of progressive...” at line 468 does not seem to belong to the previous section titled “Multimodal Interaction”; it appears that a title is missing here.
7. Authors are suggested to add experiments similar to Figure 3 in the experiment section, using Quality, Perception, Joint and proposed methods to finetune the model separately, and test it on two EIQA tasks, in order to verify that the proposed method can "improve synergy between the two EIQA tasks and enhances the multifaceted explanations of IQA".