We thank the reviewer for valuable feedback and for appreciating the idea of our method. Following are the responses regarding your concerns.

W1: The adaptation performance does not consistently improve with more unlabelled target samples.

As demonstrated in Figure 4 in the manuscript, our method employs uncertainty-aware pseudo-labels to effectively reduce the influence of noisy supervision in low-resource scenarios, outperforming naive self-learning baselines. When adapting with 100–200 unlabelled target videos, URPA achieves strong performance by filtering unreliable predictions and focusing on confident supervision. However, as the number of target samples increases, the absolute number of noisy pseudo-labels also grows, their accumulation can lead to overfitting on incorrect guidance, thereby degrading performance. This highlights a key insight: URPA is best suited for data-efficient adaptation, where the pseudo-label quality can be more reliably controlled via uncertainty estimation. Designing more robust noise-aware mechanisms for scaling to larger datasets remains an important direction for future research.

W2: How to address semantic misalignment between language queries and video content?

While our method does not introduce an explicit semantic alignment loss, the use of temporal IoU as the task reward indirectly encourages the model to align the query semantics with relevant video segments. Accurate grounding requires the model to match language cues with visual content, and the reward function provides learning signals that reinforce such alignment during adaptation.

Q1: How can models handle ambiguous or vague language descriptions?

Ambiguous or vague queries pose a major challenge for temporal grounding, as they lack explicit visual cues. Our method addresses this in two ways. First, by leveraging multiple GRPO rollouts, the model is encouraged to explore diverse temporal hypotheses rather than committing to a single point estimate. This helps uncover plausible alignments even under semantic uncertainty. Second, our use of a variance-based uncertainty metric allows the model to down-weight pseudo-labels generated from ambiguous queries, reducing their influence during adaptation. Together, these strategies help the model remain robust in the presence of vague or underspecified language.

We are grateful for the positive response, as well as the valuable suggestions. Following are the responses regarding your concerns.

W1/Q1: Whether URPA can generalize to other VLM-based temporal grounding models?

To demonstrate the generality of URPA, we conducted additional experiments using two different temporal grounding backbones: LLaVA-ST and Qwen2.5-3B. As shown in the table below, applying URPA to LLaVA-ST yields substantial performance improvements, indicating that our adaptation strategy is effective across different architectures. Similarly, URPA also improves the performance of Qwen2.5-3B, though the overall results are lower than those of the 7B model due to its more limited reasoning capacity. These results confirm that URPA generalizes well across various vision-language models, regardless of model size or backbone design.

W2/Q2: Do other unsupervised domain adaptation baselines also benefit from training on a limited subset of the target dataset?

To address the reviewer’s concern, we compare URPA with UDA-TSL, the current SOTA method for cross-domain temporal grounding, using the same limited number of unlabelled target samples. We evaluate three settings:

• UDA-TSL (NoUDA): trained only on the source domain;
• UDA-TSL (200-shot): adapted using 200 unlabelled target videos;
• UDA-TSL (2000-shot): adapted using 2000 target videos;

As the original code is unavailable, we reimplemented UDA-TSL following the paper. Results show that UDA-TSL achieves notable improvements only with ≥2000 samples. In contrast, its 200-shot variant barely outperforms the source-only baseline due to insufficient target domain coverage.

URPA, designed for data-efficient adaptation, yields substantial gains with just 200 samples and outperforms UDA-TSL in both transfer directions. This highlights URPA’s robustness and practicality for real-world low-resource scenarios where collecting large-scale unlabelled target data is infeasible. These results confirm that URPA is more suitable for few-shot unsupervised cross-domain adaptation, offering better performance without relying on extensive target data.

W3/Q3: How does URPA perform on the source domain after adapting to the target domain?

We use mIoU to evaluate source-domain performance before and after adaptation with 200-shot target data. As shown in the table below, URPA achieves better alignment with the target domain after adaptation, while source-domain performance decreases slightly as source-specific knowledge is partially replaced by target-domain features. This trade-off is typical in domain adaptation, yet URPA continues to maintain strong temporal grounding ability on the source domain.

Q4: How was the value of α selected?

We determine α through cross-validation. With small values, the model retains accurate temporal grounding while modestly correcting annotator subjectivity. Larger values further reduce annotation bias but simultaneously lower label precision. We observe that α = 0.1 provides the best balance between mitigating annotation bias and preserving temporal grounding accuracy.

We thank the reviewer for the positive evaluation of our work and our detailed feedback to the concerns are as follows.

W1(1): The necessity of efficiency for UCTG is inadequately justified.

The efficiency of UCTG is particularly critical not only because collecting dense temporal annotations is costly, but also due to the inherent storage and computational demands of video data. Compared to images, videos require significantly more space and time to annotate and store. A single video often exceeds the size of an entire image dataset, making full-set target adaptation impractical in online or real-time settings. Thus, efficient, data-lite adaptation becomes a necessity to enable scalable deployment of temporal grounding models in real-world applications such as surveillance or instructional content, where labelled data is limited and fast domain adaptation is required.

W1(2) What are the task-specific key challenges constraining UCTG development?

The development of UCTG faces three task-specific challenges. First, the lack of ground-truth supervision in the target domain makes it difficult for the model to correct domain-specific errors under domain shift. Second, temporal annotations (i.e., start and end timestamps) are often manually labeled and inherently subjective, introducing annotator bias. Finally, the duration and temporal location of events vary significantly across datasets. Without a deeper understanding of the causal or semantic structure of events, models tend to rely on learning dataset-specific temporal occurrence patterns. As a result, their predictions often fail to transfer across domains with different event distributions, leading to substantial performance drops.

To address these challenges, first, URPA introduces a confidence-based pseudo-labeling strategy to mitigate the absence of target supervision. Second, It uses a relaxed tIoU reward to reduce sensitivity to annotation bias. Thirdly, by leveraging GRPO-style reinforcement learning, the model learns to reason about event structure and temporal logic, rather than memorizing dataset-specific temporal distributions, enabling better generalization across domains.

W2: Explain the performance gaps between URPA and full-dataset methods methods in most settings, despite URPA outperforming them in a few cases?

The gap is explained by two factors. First, the richness of source-domain supervision provides different levels of grounding priors, which makes adaptation easier in some transfers such as TACoS→Charades. Second, URPA relies on GRPO-based adaptation with uncertainty filtering, which uses limited target samples more effectively but cannot fully exploit the entire target distribution. As a result, URPA can sometimes surpass full-set methods when source priors are strong, but shows gaps in transfers with larger domain differences.

W3: How does its performance evolve with increased samples?

We include an experiment in Appendix A.1 (Fig. 4) to evaluate how performance evolves with increasing adaptation samples. URPA consistently outperforms the baseline under all shot settings, confirming the effectiveness of our uncertainty-guided strategy. Interestingly, performance peaks around 300 samples and then declines. This is due to the error accumulation of noisy pseudo labels. Although our method mitigates their effect via uncertainty filtering, fully unsupervised adaptation still suffers when label quality degrades at scale. This highlights a key challenge: more unlabelled data does not necessarily yield better performance unless pseudo-label reliability is also improved. We discuss this in our limitations and plan to address it in future work.

W4: An error in Equation 3.

Thank you for pointing this out. We will correct it to reflect the intended behaviour: a reward of 1 when the output matches the required format, and 0 otherwise.

Dear Reviewer VEkt,

We hope our rebuttal has clarified your concerns. Please kindly let us know if everything looks good on your side.

Best,

We thank the reviewer for the positive evaluation of our work and for acknowledging the value and insight of the proposed method.

W1: Compare GRPO rollouts with other reinforcement learning approaches.

In our setting, such a comparison is not directly applicable. URPA is designed for unsupervised adaptation in the target domain, where no target temporal annotations are available. In contrast, most RL methods like PPO, A2C, or SAC operate in supervised or reward-accessible settings and generate rollouts across different samples, not multiple diverse predictions for the same input, which is essential for estimating epistemic uncertainty in our framework.

We compare URPA with existing reinforcement learning-based temporal grounding methods in Table below. Prior approaches such as BAR [1], TripNet [2], MBAN [3], URL [4], and the original GRPO are all trained in a fully supervised manner using the Charades dataset. In contrast, URPA is designed for unsupervised cross-domain adaptation: it is pretrained on source datasets such as TACoS or ActivityNet, and adapted to the target domain using only 200 unlabelled videos without timestamp annotations.

Despite operating without ground-truth supervision in the target domain, URPA achieves comparable or even superior performance on the Charades dataset in terms of R@0.5 and R@0.7. These results demonstrate the effectiveness of our GRPO-based uncertainty-guided adaptation strategy and its strong generalisation ability under low-resource conditions.

[1] Wu, Jie, et al. "Reinforcement learning for weakly supervised temporal grounding of natural language in untrimmed videos." Proceedings of the 28th ACM International Conference on Multimedia. 2020.

[2] M. Hahn, A. Kadav, J. M. Rehg, and H. P. Graf, “Tripping through time: Efficient localization of activities in videos,” in BMVC, 2020.

[3] X. Sun, H. Wang, and B. He, “Maban: Multi-agent boundary-aware network for natural language moment retrieval,” IEEE TIP, vol. 30, 2021.

[4] Y. Zeng, D. Cao, S. Lu, H. Zhang, J. Xu, and Z. Qin, “Moment is important: Language-based video moment retrieval via adversarial learning,” ACM TMCCA, vol. 18, 2022.

Q1: How does the choice of the exponential decay function in Eq. (9) affect the confidence score? Have alternative formulations for scaling uncertainty been tested?

The exponential decay function in Eq. (9) is designed to more aggressively suppress high-uncertainty pseudo-labels, offering sharper penalisation than linear or reciprocal alternatives and leading to more stable training. We experimented with softmax-based, sigmoid-based, and MLP-based alternatives. However, softmax requires batch-relative normalization and sigmoid compresses the variance range too aggressively, both leading to unstable training. The MLP variant adds extra computational cost while yielding inferior performance, making exponential decay a more effective and efficient choice.

W2/Q2: Does the variance metric still reliably estimate epistemic uncertainty under large domain gap?

In scenarios with significant domain shift, we acknowledge that low variance does not always guarantee correctness due to potential systematic bias from the source model. However, our method mitigates this risk in two ways. First, GRPO's structured rollouts capture epistemic uncertainty more robustly than neuron-level dropout by exposing variability in the model's temporal predictions. Second, our adaptation framework avoids relying solely on absolute variance values. Instead, we apply a decaying weighting function to softly down-weight uncertain pseudo-labels, preventing overfitting to potentially biased predictions. Empirically, as shown in the Charades→ActivityNet setting (with large visual-semantic shift), our method still achieves substantial improvements over the source training model (R@0.5: 36.5 → 42.6), confirming the robustness of the uncertainty signal.

W3/Q3: More insight into why increasing rollouts beyond 8 does not yield better results?

We observe that increasing the number of rollouts initially improves performance by stabilizing pseudo-label estimation and better capturing uncertainty (e.g., from G=4 to G=8). However, as G continues to grow, performance degrades due to limitations of the variance-based confidence metric. When most predictions are concentrated, the impact of a few divergent samples becomes diluted, causing the variance to shrink and underestimating uncertainty. This leads the model to overestimate pseudo-label reliability, weakening its ability to distinguish between reliable and unreliable supervision and ultimately hindering adaptation.

Q4: Comparison with full-set fully supervised target training results.

Although our method is few-shot and unsupervised in the target domain, it achieves strong performance compared to fully supervised target-domain training. We further compare URPA with recent state-of-the-art supervised grounding models including HawkEye [1], TimeChat [2], TRACE [3], VideoChat-T [4] and target fully supervised training GRPO, all of which are fully trained on the target Charades dataset. In contrast, URPA is pretrained on source datasets such as TACoS or ActivityNet and adapted using only 200 unlabelled Charades videos. Despite this data-efficient and unsupervised setting, URPA achieves comparable or even superior performance, highlighting its practical value and effectiveness in low-resource cross-domain scenarios.

[1] Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text llms for grounding text in videos, 2024.

[2] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313–14323, 2024.

[3] Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video llm via causal event modeling. arXiv preprint arXiv:2410.05643, 2024.

[4] Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702, 2024.

Dear Reviewer noQ9,

We hope our rebuttal has clarified your concerns. Please kindly let us know if everything looks good on your side.

Best,

We thank the reviewer for the careful reading and insightful comments. Following are the responses regarding your concerns.

W1/Q1: Could the authors include comparisons with more recent methods?

Our compared UDA-TSL (2024) is the current state-of-the-art method for cross-domain temporal grounding. To further validate URPA, we compare our URPA with recent supervised grounding models including HawkEye [1], TimeChat [2], TRACE [3] and VideoChat-T [4] on Charades dataset , all of which are fully trained on the Charades dataset. In contrast, URPA is pretrained on source datasets such as TACoS or ActivityNet and adapted using only 200 unlabelled videos from Charades. Despite this data-efficient and unsupervised setting, URPA achieves comparable or even better performance, highlighting its strong effectiveness in low-resource scenarios.

[1] Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text llms for grounding text in videos, 2024.

[2] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313–14323, 2024.

[3] Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video llm via causal event modeling. arXiv preprint arXiv:2410.05643, 2024.

[4] Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702, 2024.

W2/Q3: What constitutes the core methodological novelty beyond integration?

Our core methodological novelty lies in reinterpreting the rollout variance from GRPO as a principled uncertainty signal for test-time adaptation. Unlike MC Dropout, which injects randomness at the neuron level, our approach leverages policy-level stochasticity to estimate epistemic uncertainty without requiring architectural changes. This insight enables the first unsupervised reinforcement learning–based temporal grounding framework, where uncertainty guides adaptation without any temporal annotations. Furthermore, we formalise and address a new and practical setting of data-efficient unsupervised cross-domain temporal grounding, which remains underexplored in prior work.

W3/Q2: Whether URPA can generalize to other VLM-based temporal grounding models?

To demonstrate the generality of URPA, we conducted additional experiments using two different temporal grounding backbones: LLaVA-ST and Qwen2.5-3B. As shown in the table below, applying URPA to LLaVA-ST yields substantial performance improvements, indicating that our adaptation strategy is effective across different architectures. Similarly, URPA also improves the performance of Qwen2.5-3B, though the overall results are lower than those of the 7B model due to its more limited reasoning capacity. These results confirm that URPA generalizes well across various vision-language models, regardless of model size or backbone design.

Thank you again for your thoughtful review. Could you kindly let us know if our response addressed your concern? Your feedback would be very helpful for us to better understand your evaluation and improve our paper.

Dear Reviewer vGBE,

As the author–reviewer discussion phase is ending soon, and we cannot view your final score or justification, we’d greatly appreciate it if you could briefly let us know whether our rebuttal addressed your concerns.

Best, The authors