AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.

1. Definition of evaluation metrics

SRCC and KRCC measure the prediction monotonicity, while PLCC and RMSE measure the prediction accuracy. Better AGAVQA methods should have larger SRCC, KRCC, PLCC values, and smaller RMSE values. SRCC can be formulated as:

$SRCC = 1-\frac{6\sum^N_{n=1}(v_n-p_n)^2}{N(N^2-1)}$,

where $v_n$ and $p_n$ denote the rank of the MOSs and predicted score, respectively. For a pair of ranks $(v_i,p_i)$ and $(v_j,p_j)$, the pair is concordant if $(v_i-p_i)(v_j-p_j)>0$, and discordant if $<0$. KRCC is defined as:

$KRCC=\frac{C-D}{N(N-1)/2}$,

where $N$ is the number of AGAVs, $C$ and $D$ denote the number of concordant and discordant pairs, respectively. PLCC and RMSE are calculated as:

$PLCC = \left(\sum^N_{n=1}(y_n-\overline{y})(\widehat{y}_n-\overline{\widehat{y}})\right)/$

$\left(\sqrt{\sum^N_{n=1}(y_n-\overline{y})^2\sum^N_{n=1}(\widehat{y}_n-\overline{\widehat{y}})^2}\right)$,

$RMSE = \sqrt{\frac{1}{N} \sum_{n=1}^{N} (y_n - \hat{y}_n)^2}$,

where $y$ and $\widehat{y}$ denote the MOSs and predicted scores, respectively. $\overline{y}$ and $\overline{\widehat{y}}$ are the mean of the MOSs and predicted scores, respectively.

2. Introduction to TTA and TTM datasets

TTA and TTM were proposed by [5]. The TTA dataset generates 500 audio samples from 100 prompts using 5 text-to-audio generation methods. Subjects were then invited to rate the quality of the audio and its relevance to the provided description. Similarly, the TTM dataset generates 500 music samples from 100 prompts using 5 text-to-music generation methods. Subjects were asked to rate the quality of the music and its relevance to the provided description.

3. More experiments

On the AGAVQA-Pair subset, AGAV-Rater is tested using a single-input scoring approach. We present the accuracy of fine-tuned audio-video alignment methods on the AGAVQA-Pair subset and the accuracy of random selection:

As can be seen, AGAV-Rater achieves the highest accuracy in each category.

4. Distribution of AIGC videos in section 5.6

Thank you for your suggestion. We statistically analyze the distribution of the 230 AIGC video contents in Section 5.6 (in the manuscript), and the results are as follows:

As shown in the table, the video content is quite diverse, allowing for a comprehensive evaluation of AGAV-Rater's performance across different types of video content.

5. Explanation of standard text quality levels

Researchers have found that "good" and "poor" are the most frequently predicted tokens by LMM models when addressing quality issues. We then use the standard text rating levels defined by ITU [6]—excellent, good, fair, poor, and bad—to further refine the quality levels corresponding to these tokens.

References

[1] Hayes, A. F. and Krippendorff, K. Answering the call for a standard reliability measure for coding data. Commun. Methods Meas., 1(1):77–89, 2007.

[2] Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In CVPR, pp. 26584–26595, 2024

[3] Li, Z., Xu, Q., Zhang, D., Song, H., Cai, Y., Qi, Q., Zhou, R., Pan, J., Li, Z., Tu, V., et al. Groundinggpt: Language enhanced multi-modal grounding model. In ACL, pp. 6657–6678, 2024.

[4] R. I.-R. BT, Methodology for the subjective assessment of the quality of television pictures. ITU, 2002.

[5] Deshmukh, S., Alharthi, D., Elizalde, B., Gamper, H., Ismail, M. A., Singh, R., Raj, B., and Wang, H. Pam:Prompting audio-language models for audio quality assessment. arXiv preprint arXiv:2402.00282, 2024.

[6]Recommendation 500-10: Methodology for the subjective assessment of the quality of television pictures. ITU-R Rec.BT.500,2000.

We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.

1. Correlation of 3-dimensional MOSs

SRCC between audio quality and content consistency is 0.6860, indicating that the two dimensions are independent. SRCC between audio quality and overall quality is 0.7876, and between content consistency and overall quality is 0.7926, suggesting that overall quality is influenced by both audio quality and content consistency.

2. Differences between AGAV-Rater and audio-video alignment methods

The key difference between AGAV-Rater and VAST is that AGAV-Rater utilizes the semantic understanding ability of LLM, improving performance. Although audio-video alignment methods align features semantically, we fine-tuned them on the AGAVQA-MOS subset, mapping audio and video features to quality dimensions. After fine-tuning, these features contain both alignment information and quality information. Therefore, the suboptimal performance of alignment methods does not imply that AGAVQA-MOS emphasizes alignment.

3. More experiments

We conduct ablation studies to compare AGAV-Rater with fine-tuned LMMs, with a detailed analysis in Section 3 of the response to Reviewer R3k7. We also test on the AVQA dataset, SJTU-UAV, focusing on real-world user capture distortions.

AGAV-Rater achieves the best performance on SJTU-UAV, proving its superiority comes from the model framework and training, not from dataset adaptability.

4. Details of audio processing

AGAV-Rater uses default parameters from VideoLLaMA2 for audio preprocessing. Assuming $T$ video frames are extracted, the steps are:

1. Divide audio into $T$ segments.
2. Concatenate all segments, then crop or zero-pad to a fixed length.
3. Transform into fbank spectrograms with 128 frequency bins.
4. Use BEATs and an MLP block to extract features from the spectrograms.
5. Concatenate audio, video, and text features, and input them into the LLM.

5. Details of human evaluation

We invited subjects familiar with AVQA and AGAV for on-site training. We provided detailed explanations of the scoring criteria for each dimension and additional AGAV samples for practice. Experts then reviewed the annotations and selected 15 subjects. To prevent fatigue, each subject rated a maximum of 60 samples per day, completing the task in about two months.

We used the ITU-recommended MOS processing method [4], and no subjects were identified as outliers. Krippendorff's α [1] for audio quality, content consistency, and overall quality are 0.6814, 0.7343, and 0.7143, respectively, indicating appropriate variations among subjects. We also randomly divide subjects into two groups and calculate the SRCC of average scores between the two groups. After ten repetitions, the average SRCC for audio quality, content consistency, and overall quality are 0.8043, 0.8318, and 0.8297, validating rating consistency.

6. Supplement to the AGAVQA-Pair subset

Due to the lack of public AGAVQA datasets, the AGAVQA-Pair subset was collected from 8 VTA webpages released in the past year. Its significance lies in that, as a third-party platform, it offers a more objective and impartial dataset.

Best-of-pair selection is more reliable than scoring tasks. Subjects show greater consistency and confidence in determining the optimal sample. In practical applications, identifying the optimal AGAV may be enough. We use 230 AGAV pairs collected in Section 5.6 (in the manuscript) to further validate generalization. The accuracy of AGAV-Rater, along with fine-tuned VAST, VALOR, and AVID-CMA, is 80%, 74%, 69%, and 68%, respectively.

7. Modification of Fig.2

We apologize for the error in Fig. 2. There are 45 Kling video sources in total, with 23 from the Kling official website, and 22 from the video generation benchmark Vbench. This will be corrected in the final manuscript.

8. Motivation of our paper

Previous AVQA and AQA work focused on real-world capture or compression distortions. Audio-video alignment methods targeted semantic alignment in real scenarios. LMM-based quality assessment primarily centered on visual quality, with a limited focus on audio. As AIGC video technology advances, more research explores dubbing techniques. The motivation of our paper is that the quality of AGAVs needs to be monitored and controlled. In the AGAVQA-MOS subset, both audio and video content are AI-generated. We aim to use LMMs to evaluate AGAV quality, replacing human subjective scoring to enhance efficiency and enable automation.

9. Correction of grammar errors

Thank you for pointing out our grammatical errors. We will correct these mistakes in the final manuscript.

References

Please refer to the Response to Reviewer h1V5.

We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.

1. Details of the auto-labeling process

We manually verify 500 auto-labeling results. Among them, the accuracy for content consistency related instruction-response pairs is $100$%, while the accuracy for audio quality related instruction-response pairs is $92$%. In content consistency-related instruction-response pairs, when the consistency quality is labeled as "bad", we ensure that audio (text) and video from different categories are paired to achieve high accuracy. In audio quality-related instruction-response pairs, for noisy audio types, such as machine sounds or wind noise, the reverse operation has a minimal negative impact on audio quality. We have utilized category labels to filter out certain audio quality-related instruction-response pairs, such as hair dryer drying and pumping water, to minimize the error rate.

2. Anlysis of human scores

The range of MOSs is $[0,100]$. We categorize the video content into 11 main audio sources and then calculate the standard deviation of the overall quality scores among 15 subjects. We compute the average standard deviation for each category:

The "Fantasy" category shows the highest standard deviation, as it represents unreal scenarios, leading to more diverse interpretations among subjects. Krippendorff's α [1] can be used to measure the quality of the subjects' ratings. We calculate Krippendorff's α for audio quality, content consistency, and overall quality, which are 0.6814, 0.7343, and 0.7143, respectively, indicating appropriate variations among subjects. We also randomly divide subjects into two groups and calculate the SRCC of average scores between the two groups. After ten repetitions, the average SRCC for audio quality, content consistency, and overall quality are 0.8043, 0.8318, and 0.8297, validating rating consistency.

3. Ablation study of the base models

We conduct ablation studies using GroundingGPT [2] and OneLLM [3] as base models on the AGAVQA-MOS subset. For OneLLM and GroundingGPT, we first load the default weights, and then fine-tune them using the official training code on the AGAVQA-MOS subset. To ensure fairness, we also add the quality regression module, directly regressing the LLM's last hidden states to output three-dimensional numerical scores. The results are as follows:

As shown in the experimental results, AGAV-Rater achieves the best performance. The main reason for this is that VideoLLaMA2 is designed for audio-video content understanding and pre-trained on more diverse audio-video datasets, making it more suitable for our quality assessment task. GroundingGPT focuses more on localization and visual understanding and is not designed or trained to understand continuous audio-video content. Its ability to comprehend video quality may be weaker. OneLLM is a general multimodal model that, while supporting audio and video processing, is not specifically optimized or enhanced for video and audio alignment. Its audio-related dataset only includes audio-text data, and OneLLM is more suited to text-vision or text-audio matching and understanding, rather than specific audio-video content.

4. More cases displays

Thank you for your suggestion. We have displayed more cases about the samples and the corresponding scores rated by the AGAV-Rater on the project page (https://agav-rater.github.io).

References

Please refer to the Response to Reviewer h1V5.

We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.

1. Details of human evaluation

We invited subjects familiar with AVQA and AGAV for on-site training. We provided detailed explanations of the scoring criteria for each dimension and additional AGAV samples for practice. Experts then reviewed the annotations and selected 15 subjects. To prevent fatigue, each subject rated a maximum of 60 samples per day, completing the task in about two months.

The official testing phase was conducted in a controlled lab with normal indoor lighting, quiet surroundings, and subjects sitting at a comfortable distance of about 60 cm from the screen. The AGAVs were played at their original resolution. The scoring interface consisted of three continuous quality rating bars and three navigation buttons. Each rating bar was labeled with a 1-5 Likert scale. Navigation buttons, including "Prev", "Repeat", and "Next", allowed subjects to switch and replay AGAVs freely. In the final manuscript, we will add images of the scoring interface and detailed documentation of the scoring criteria provided to subjects.

2. Training duration

We trained AGAV-Rater on two 96GB H20 GPUs, with training epochs set to $5$ on the AGAVQA-MOS subset, taking approximately $5$ hours.

3. Details of audio-video alignment methods

Original audio-video alignment methods extract audio and video features using their encoders, then align them into a common vector space. We use these encoders with default parameters to extract audio and video features and then concatenate features. The concatenated features are fed into a fully connected layer with an output dimension of 3 to predict three-dimensional scores.

4. Distribution of AIGC videos in Section 5.6

Due to word limitations, the distribution of AIGC videos is described in Section 4 of the response to Reviewer h1V5. We apologize for any inconvenience.

5. Modification of Fig. 2

Thank you for your suggestion. In the final manuscript, we will add (a), (b), and (c) to Fig. 2 to help readers quickly understand its content.

6. Introduction to the loss function

We use the PLCC loss to optimize the AGAV-Rater:

$L=(1-\frac{\left<\widehat{s}-mean(\widehat{s}), s-mean(s)\right>}{\lVert\widehat{s}-mean(\widehat{s})\rVert_2\lVert s-mean(s)\rVert_2})/2$,

where $s$ and $\widehat{s}$ are the vectors of MOSs and predicted scores of AGAVs in a batch respectively, $\left<\cdot\right>$ represents the inner product of two vectors, $\lVert\cdot\rVert$ denotes the norm operator for a vector, and $mean$ is the average operator for a vector.

7. Details of instruction-response pairs

In the 50,952 instruction-response pairs, the audio-video, audio-text, and music-text scenarios contain 25,592, 19,000, and 6,000 pairs, respectively. Under each scenario, half of the pairs focus on content consistency, and the other half on audio quality.

8. Analysis of Table 4

AGAV-Rater uses VideoLLaMA2 as the base model, and the training set used by VideoLLaMA2 mainly focuses on audio, with relatively less on music. In Table. 2 (in the manuscript), it can be seen that its ability to perceive music-text quality is weaker compared to audio-video and audio-text. Although the music-text scenario has the fewest instruction-response pairs, we repeat these pairs twice during pre-training to increase the learning times of AGAV-Rater for the music-text scenario. Therefore, the music-text instruction-response pairs enhance the music perception ability of AGAV-Rater, improving the performance of consistency dimension on the TTM dataset.

9. Supplement to Table 3

The AGAVQA-Pair subset was collected from 8 VTA webpages, dividing it into 8 corresponding categories. For each category, the optimal AGAV is generated by the corresponding VTA method. We chose to collect the AGAVQA-Pair subset from VTA webpages because these AGAVs, sourced from third-party platforms, offer a more objective and impartial dataset. These VTA webpages are all released in the past year, representing the latest technology in VTA methods.

The compared methods in Table 3 use their original model parameters without fine-tuning on the AGAVQA-MOS subset. We further show the accuracy of the audio-video alignment methods which have been fine-tuned on the AGAVQA-MOS subset:

As can be seen, AGAV-Rater achieves the highest accuracy in each category.