RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts

We sincerely thank Reviewer 6kbe for the constructive comments and suggestions on RadarQA. We are glad that the completeness and versatility of our four-task paradigm was appreciated. We are encouraged by the reviewer's recognition of our strong performance in both in-distribution and out-of-distribution settings. We address the reviewer's comments in more detail below.

Q1: Potential bias and cost of using GPT-4-generated labels.

Thank you for the reviewer’s suggestion.

First, the key attributes of our dataset is annotated by meteorological experts or calculated by meteorological metrics. GPT-4o is only used to organize these attributes into a fluent language. Therefore, the potential bias introduced by GPT-4o is very small.

Second, the total cost of GPT-4o labeling was approximately $700. We will include a discussion of the limitations associated with GPT-4o annotations in the final version of the manuscript.

Q2: Blind human review.

We conduct human experts evaluation in the "Expert study" section (Page 9, Line 326-332) of the submission, where our method show clear advantage over GPT-4o. Specifically, we first randomized the ground truth, RadarQA’s outputs, and GPT-4o’s outputs. Then, we invite nine meteorology experts to participate in the expert study. Based on the observed images, model predictions, and different quality assessment reports, the experts considered the following three dimensions to provide an overall score: (1) Accuracy of the textual content, (2) Whether the issues of concern to the experts are mentioned in the report, (3) The density of effective information.

Q3: Reproducibility challenges due to reliance on proprietary models and high computational cost.

First, we will release all codes, collected datasets, and model weights for reproducibility.

Second, we highlight that fine-tuning a 7B-parameter model is not so costly, and can be reliably performed on 2 NVIDIA RTX 4090 GPUs, with inference requiring only a single NVIDIA RTX 4090 GPU.

Third, we also trained a smaller 3B model by using Qwen2.5-VL-3B-Instruct as the base model under the same training settings. The results are given below.

Q4: Comparison with stronger domain-specific baselines.

We compare RadarQA with domain specific baselines in the table below. The results show that RadarQA significantly outperforms all selected baselines, highlighting its effectiveness in aligning with human experts.

The domain specific baselines are: Weather specific metrics including CSI, POD, FAR, bias, and ETS, with thresholds set at 74 and 160, and 181, and General FR-IQA methods including PSNR, SSIM, LPIPS and DISTS.

Q5: Ethical and social impact discussion.

We appreciate the suggestion. We acknowledge that weather prediction assessment may have a quite huge influence in humans' daily life and production. We will include a dedicated caution section addressing these considerations in the final version to ensure responsible use and awareness of the limitations of our work.

We sincerely thank Reviewer 6kbe for recognizing our work. We will revise our manuscript carefully according to your suggestions. Below are more details about Q1 and Q2.

Q1: Potential bias and cost of using GPT-4-generated labels.

We use GPT-4o to organize annotated attributes into assessment descriptions for assessment tasks, which may introduce potential bias, including:

1. Style bias. The structure of generated reports might be similar can cannot reflect the expert diversity.
2. accuracy bias. The generated content may not fully consistent with the visual information in the label.
3. Redundancy bias. GPT-4o may generate descriptions that contain redundant information, which often reduces clarity of the evaluation.
4. Attribute Omission bias. GPT-4o may ignore less prominent yet important features, resulting in incomplete assessments.

We will include a more detailed discussion of the potential biases introduced by GPT-4o in the Limitation section, covering issues such as style bias, accuracy bias, and others identified above.

Q2: Blind human review.

We appreciate the reviewer’s insightful suggestion. We will carry out a larger-scale human validation and expert study to further support our findings, we are currently limited by time and thus unable to provide more accurate and comprehensive results at this stage.

We sincerely thank Reviewer Htbn for detailed feedback. We greatly appreciate the recognition of our clearly motivated problem setting, dataset construction strategy. We are glad that the reviewer found our four-task matrix well aligned with real-world forecasting practices. Below, we address the reviewer's main concerns point by point.

Q1: Comparison with traditional weather analysis methods and weather-specific ML models.

First, we compare our method with meteorology-specific metrics and image quality assessment (IQA) metrics on the overall Frame Rating task in the table below. The accuracy, PLCC, and SRCC reflect the consistency between the evaluation results and the expert annotations. RadarQA significantly outperforms the baseline traditional metrics across all three evaluation metrics. While these traditional metrics are reasonable, there remains a noticeable gap between their outputs and expert judgment. These baseline metrics are meteorology-specific ones, including CSI, POD, FAR, bias, and ETS (with thresholds 74/160/181), and IQA metrics including PSNR, SSIM, LPIPS, and DISTS.

Second, to the best of our knowledge, there are no existing works in the meteorological domain that apply machine learning models for quality evaluation. Traditionally, meteorological forecast quality assessment relies on two approaches: expert subjective judgment and objective metric-based scoring.

1. Expert subjective judgment requires high costs and specialized domain knowledge.
2. Objective metric-based scoring cannot comprehensively evaluate forecast results, lacking understanding and assessment of proprietary meteorological patterns (e.g., convective systems).

Therefore, we leverage the power of MLLMs to effectively integrate expert knowledge (text-based quality assessments) with objective metrics (level-based quality rating) for quality evaluation, enabling a comprehensive assessment that combines domain expertise and quantitative indicators at a lower cost.

Q2: RL training details.

We appreciate the reviewer's question. We have provided descriptions of the RL training details in the main paper (Page 7, Lines 258-269). We will elaborate on it below for completeness.

Overall RL method. In the RL post-training stage, we adopt the GRPO algorithm to optimize the model's performance on the rating task. Given a query $q$, the model with old parameters $\pi_{old}$ generates $N$ different responses ${o_1, o_2, ..., o_N}$. Then, a custom reward model is used to compute the corresponding rewards ${r_1, r_2, ..., r_N}$. Unlike other online RL methods (e.g., PPO), GRPO utilizes group-wise averaging to estimate the relative advantage without requiring an additional value model. The computation is performed as follows:

$$A_i=\frac{r_i-\text{mean}(\{r_1,r_2,...,r_N\})}{\text{std}(\{r_1,r_2,...,r_N\})}$$

Based on the advantage function, the objective function can be expressed as:

$$\mathcal{J}(\theta)=\mathbb{E} \left[ \min(\rho_i A_i,\ \text{clip}(\rho_i, \ 1 - \delta, \ 1 + \delta)\ A_i) - \beta \cdot D_{\mathbb{KL}}(\pi_{\theta_{new}}||\pi_{ref})\right]$$

Here，$\rho_i=\frac{\pi_{\theta_{new}}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}$, $q\sim{Q}$ and $o_i\sim\pi_{\theta_{old}}$.

Format reward. The format reward function uses regular expressions to determine whether the model's output can be correctly parsed. If the format of output can be successfully parsed into a JSON format and all keys exactly match those in the ground truth, the reward is 1; otherwise, the reward is 0.

Accuracy reward. Let $n$ be the total number of keys in the ground truth (i.e. n = len(gt_dict.keys())). If the extracted answer has $k$ keys whose values match those in the ground truth, then the reward is $k/n$. Since RL training is only applied in rating tasks where the answer could be "poor", "fair", "good", and "great", it is easy to compare the predicted answer to the ground truth. The output is first parsed into a valid JSON format to extract the answer. If the parse fails, the reward is also 0.

Number of training epochs. A shown in the table below, training with 1 epoch has led to good enough performance, and increasing the number of training epochs does not lead to significant performance improvements. Therefore, we choose to train for 1 epoch to balance computational cost and model performance.

Q3: Labeling in Figure 2.

We appreciate the reviewer's suggestion. In Figure 2, we use different colors to highlight some key information. We will simplify the use of colors to make it easier to read. We will also add a label to improve clarity and understanding.

We sincerely thank Reviewer 78LW and the area chair for their efforts and appreciation of our comprehensive dataset analysis for the weather forecasting task, the diversity of question types, and input modalities. We will update the manuscript as suggested. Below, we address the reviewer's main concerns point by point.

Q1：Contributions beyond dataset collection.

In fact, the lack of a large-scale, high-quality dataset is the main gap in the field of quality evaluation for nowcasting tasks. Recently, many MLLM-based assessment methods, such as Q-Instruct[1], Co-Instruct[2], and DepictQA[3], have also emphasized the importance of dataset construction for quality evaluation. In the field of meteorology, existing efforts have largely focused on developing new numerical metrics, while neglecting a quality understanding and perception grounded in expert evaluation practices. In contrast, our proposed dataset, RQA-70K, addresses this gap by incorporating both frame-level and sequence-level data modalities and supporting both rating-based and assessment-based evaluation tasks, thereby bridging numerical evaluation metrics and expert-driven analysis approaches.

Besides the collected dataset, our contributions mainly consist of three aspects:

1. A novel quality evaluation paradigm for meteorology. By integrating the evaluation steps of meteorological experts and existing metric-based quality assessment methods, we propose four quality analysis tasks: frame rating, frame assessment, sequence rating, and sequence assessment. These tasks cover both frame-level and sequence-level modalities, as well as rating-based and assessment-based evaluation needs.
2. A well-designed data construction pipeline. To construct our dataset, we begin by selecting seven classical nowcasting models based on various backbones to generate the RawRQA-20K dataset by using storm events from SEVIR. Second, in collaboration with domain experts, we establish a scientific attribute library containing a total of 37 attributes. Based on the characteristics of these attributes, we adopt a hybrid labeling strategy that combines automated and manual annotation on RawRQA-20K. Finally, we carefully design instruction prompts to guide GPT-4o in generating data for assessment tasks based on these attributes, and we use scripts to construct JSON-formatted data for rating tasks, ultimately resulting in the RQA-70K dataset.
3. A multi-stage training pipeline. To train a high-performance quality evaluation model, we first employ supervised fine-tuning (SFT) to equip the model with foundational capabilities. Then, we adopt GRPO along with two carefully designed reward functions on the two rating tasks. Finally, we conduct an additional round of SFT on a small subset of RQA-70K with a lower LoRA rank to further enhance the model's performance.

[1] Wu H, et al. Q-Instruct: Improving low-level visual abilities for multi-modality foundation models, CVPR 2024.

[2] Wu H, et al. Towards open-ended visual quality comparison, ECCV 2024.

[3] You Z, et al. Depicting beyond scores: Advancing image quality assessment through multi-modal language models, ECCV 2024.

Q2：More foundation models.

We adopt Qwen2.5-VL-7B-Instruct as our base model, following many recent quality evaluation studies [1,2,3]. To provide a more comprehensive comparison, here we additionally conduct experiments using Qwen2.5-VL-3B-Instruct. The experimental results below show that the 3B model shows a slight drop in performance while using fewer parameters.

[1] Li W, Zhang X, Zhao S, et al. Q-Insight: Understanding image quality via visual reinforcement learning[J]. arXiv preprint arXiv:2503.22679, 2025.

[2] Wu T, Zou J, Liang J, et al. VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank[J]. arXiv preprint arXiv:2505.14460, 2025.

[3] Zhang X, Li W, Zhao S, et al. VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning[J]. arXiv preprint arXiv:2506.18564, 2025.

Dear reviewer 78LW:

Thank you again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know.

We sincerely look forward to your feedback.

We sincerely thank Reviewer AvwT and the area chair for the thoughtful feedback and appreciation for our training strategy and ablation studies. We also appreciate the positive comments on the annotation pipeline and clarity of the overall presentation. We will continue to refine the manuscript accordingly. Below, we address the reviewer's concerns in detail.

Q1：Public availability of the proposed dataset and codes.

We commit to releasing all datasets, codes, and model weights upon acceptance of the paper to support community usage and reproducibility.

Q2：Background of task attribute definition based on the scientific attributes library.

Our model is designed to evaluate the quality of short-term precipitation forecasting.

Short-term precipitation forecasting [1, 2] is one of the most fundamental tasks in the meteorological domain. In our experiments, forecasting models are tasked with predicting the next 12 frames of precipitation based on the previous 10 frames. These predictions are intended to support early warning systems and meteorological agencies in making timely decisions to mitigate the impact of severe convective weather events.

To evaluate the quality of such forecasts, meteorological experts typically rely on a combination of professional knowledge and formula-based indicators, aiming to ensure the reliability of assessments.

Inspired by this evaluation paradigm, we collaborated with a frontline meteorologist with nearly 20 years of experience to construct a scientific attribute library for assessing forecast quality from both physical and perceptual standpoints, ensuring professionalism.

Some attributes are derived from metrics, aligning with expert use of objective indicators, while others are designed based on perceptual and domain-specific meteorological knowledge, reflecting expert experience. This library allows for a comprehensive evaluation of model outputs that mirrors expert judgment. The detailed definitions and properties of all attributes can be found in the supplementary material.

[1] Gong J, Bai L, Ye P, et al. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling[J]. arXiv preprint arXiv:2402.04290, 2024.

[2] Zhang Y, Long M, Chen K, et al. Skilful nowcasting of extreme precipitation with NowcastNet[J]. Nature, 2023, 619(7970): 526-532.

Q3: Resource-intensive training strategy and reproducibility across diverse weather forecasting scenarios.

We appreciate the reviewer's question. The proposed training strategy is indeed applicable to other weather-related tasks.

First, fine-tuning a 7B-parameter model is not so costly, and can be reliably performed on two NVIDIA RTX 4090 GPUs, with inference requiring only a single GPU.

Second, smaller models can also be used depending on the application scenario. We conducted experiments under the same setting using Qwen2.5-VL-3B-Instruct in the table below. Although there is a slight performance drop, it still demonstrates promising potential for quality assessment.

Third, data is the key factor to adapt this MLLM-based method to diverse weather forecasting scenarios. Researchers can follow our proposed dataset construction pipeline to construct new datasets.

Finally, with constructed datasets, the proposed training strategy is feasible for a wide range of tasks.

Q4: Annotator profiles.

First, we invited meteorological experts to define the annotation guidelines and standards.

Second, we recruited a total of 16 annotators with undergraduate-level education or higher, all with backgrounds in meteorology and prior annotation experience.

Then, annotators were asked to conduct a pilot annotation phase, during which their results were reviewed and validated by experts.

Also, feedback from this phase was used to refine and clarify the annotation rules, aiming to minimize annotator bias and ensure consistency throughout the labeling process.

Q5: Potential information loss.

We appreciate the reviewer's concern.

First, the conversion from raw radar data to RGB image is almost lossless. We need to convert the raw radar data (float32) into RGB format (uint8). Our statistics below with RMSE, MAE, R2, PSNR, and SSIM metrics show that this conversion is almost lossless.

Second, for better visualization and to facilitate understanding of different precipitation levels, we follow previous meterological works [1,2] to divide the precipitation into seven discrete levels (i.e., sunny, light, moderate, heavy, very heavy, intense, and extreme), which is a common and standard visualization process in the meteorological field.

[1] Gong J, Bai L, Ye P, et al. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling[J]. arXiv preprint arXiv:2402.04290, 2024.

[2] Robinson M, Evans J, Crowe B. En route weather depiction benefits of the NEXRAD vertically integrated liquid water product utilized by the corridor integrated weather system[C]//10th conference on aviation, range and aerospace meteorology, american meteorological society, portland, or. 2002.

Q6：More details about the dataset samples and the fine-tuning instruction prompts.

We commit to releasing all datasets, codes, and model weights upon acceptance of the paper to support community usage and reproducibility. The system prompt and instruction-tuning prompt used in our experiments will also be fully released.

Some samples from RadarQA, along with the corresponding prompt information, are also provided in the supplementary material (Figure A7, Figure A8, and Tables A6–A9).

Q7: Detailed LoRA fine-tuning training configuration.

We appreciate the reviewer's feedback. We have provided a description of the training configuration in the main paper (Page 7, Lines 274–280). Below, we further supplement it with additional implementation details.

In the first supervised fine-tuning stage, we set lora_rank and lora_alpha set to 8 and 32, respectively, applied to all linear layers. The model is trained for 5 epochs on 8 NVIDIA A800 GPUs, with a batch size of 2 per GPU and gradient_accumulation_steps set to 8. DeepSpeed ZeRO Stage 2 is employed to optimize memory usage. We set VIDEO_MAX_PIXELS to 602112 and FPS_MAX_FRAMES to 12 during training.

In the second reinforcement learning stage, we add new LoRA parameters, with lora_rank and lora_alpha set to 8 and 32. The batch size per GPU is set to 4, with gradient accumulation steps of 8, training for 1 epoch. We utilize vLLM to accelerate inference, generating 8 samples per iteration for GRPO reinforcement learning. Training is conducted using DeepSpeed ZeRO Stage 3.

In the third post-training stage, new LoRA parameters (lora_rank=8 and lora_alpha=32) are added. We randomly select 10,000 samples from RQA-70K for a small-scale supervised fine-tuning. The training parameters are similar to those in the first stage, except that lora_rank is set to 4.

Q8: Reasons for low BLEU and METEOR scores.

We appreciate the reviewer's feedback.

First, one reason for the relatively low BLEU scores is the metric's sensitivity to exact n-gram matches. BLEU was originally designed for machine translation. As a result, if the predicted sentences use synonyms or have different word order compared to the references, BLEU may penalize them heavily, even if the semantic meaning is similar[1].

Second, the inherently challenging nature of our task contributes to lower scores across BLEU and METEOR. Each sample of the assessment tasks contains complex, informative meteorological descriptions, which makes it more difficult to achieve high scores, especially compared to less domain-specific tasks. For example, in WeatherQA[2], a domain-specific task focusing on severe weather reasoning, even after fine-tuning, MLLMs still yield relatively low BLEU and METEOR scores, similar to those in our setting. This further highlights the inherent difficulty of domain-specific tasks for current MLLMs.

Finally, despite the relatively low BLEU and METEOR scores, the overall performance of RadarQA shows consistent improvements over baselines, which demonstrates strong potential for weather quality analysis.

[1] Celikyilmaz A, Clark E, Gao J. Evaluation of text generation: A survey[J]. arXiv preprint arXiv:2006.14799, 2020.

[2] Ma C, Hua Z, Anderson-Frey A, et al. Weatherqa: Can multimodal language models reason about severe weather?[J]. arXiv preprint arXiv:2406.11217, 2024.