Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

Dear Reviewer ATYx,
Many thanks for your detailed and valuable reviews. We address your concerns one by one below.

Question 1

We appreciate the reviewer’s interest in more extensive hyperparameter ablations. Our design choices were carefully guided by a trade-off between caption quality and inference efficiency, and several related analyses are already provided in the main paper and appendix. To further address this concern, we have conducted additional experiments, summarized below:

1. Number of candidates K:
As stated in Sec. 4.1, we fixed K = 6 candidates per temperature to achieve a practical trade-off between performance and compute cost. Increasing K beyond 6 does improve quality slightly but scales inference cost linearly, making it impractical for deployment (e.g., K = 12 nearly doubles runtime with marginal gains). We provide additional results with K = 2 in Table R1, which show a clear performance drop.

2. Temperature settings:
Our multi-temperature strategy {0.1, 0.3, 0.5, 0.7, 0.9} (Sec. 4.1) was chosen to ensure diversity while maintaining grounding. In Table R1, new fixed-temperature experiments (e.g., T = 0.2) show degraded caption quality and higher hallucination rates, validating the importance of varied entropy in candidate generation.

3. Margin threshold τ:
The rationale and calibration for τ are explicitly detailed in Appendix D. We select τ at the 17–20 percentile of the CLIP similarity distribution to penalize weakly grounded sentences without over-penalizing borderline cases. This percentile-based choice ensures robustness across datasets, as shown by our stability analysis (Appendix D, lines 547–563).

Due to time constraints, we prioritized core results, but we agree more ablations (e.g., full K and τ sweeps) would be valuable and plan to include them in the camera-ready version. Preliminary tests already indicate that our method is stable across reasonable ranges of these hyperparameters.

Question 2

The failure modes and their mitigation are already discussed in multiple parts of our paper. Specifically, Section 5 (Observations and Limitations, lines 329–354) and Figures 2–5 provide qualitative analyses comparing ViMaR and VisVM, explicitly marking grounded (green) versus hallucinated (red) segments. These examples illustrate the primary failure cases of prior methods—such as misattributing visual attributes or overlooking peripheral details—and demonstrate how our two-stage refinement and margin-bas...

Importantly, our margin-aware reward directly addresses these failure cases by sharply penalizing low-confidence hallucinated segments and refining them selectively in Stage 2, avoiding costly full caption regeneration. While residual errors (e.g., in highly ambiguous scenes) are acknowledged, they occur far less frequently and are highlighted in our qualitative case studies (Appendix E).

Could the reviewer please clarify which specific failure cases they believe are missing? We would be happy to more detailed discussion of these cases.

Question 3

We thank the reviewer for pointing out the need for additional clarity regarding inference search of our two-stage inference framework. While Stage 1 identifies the highest-value caption via best-of-N generation guided by the value model $V_ρ$, Stage 2 is specifically designed to refine only visually under-grounded segments rather than regenerate the entire caption, which reduces compute overhead while improving fidelity.

1. Segment Identification: After Stage 1 generates the coarse caption y*, we compute sentence-level value scores for each segment y_i relative to the image. Segments falling below the calibrated margin threshold 1.87 (similarly calculated as τ) are flagged as “under-grounded” and removed from the sentence. Additionally, missed visual entities (e.g., objects detected in image features but absent from the caption) are flagged via cross-attention coverage analysis.

2. Candidate Generation: For each flagged segment, we condition decoding on the preceding accepted segments y_{<i} and image I, sampling K = 6 candidates per temperature from the fixed set {0.1, 0.3, 0.5, 0.7, 0.9} (same as Stage 1). These candidates are short continuations targeted to replace only the flagged segment.

3. Scoring and Selection: Each candidate is scored using the value model $V_ρ(s,I)$, which is trained using temporal-difference value estimation (Sec. 3.1). The highest-value candidate replaces the flagged segment in the caption.

4. Iterative Refinement and Termination: This process iterates until all flagged segments exceed the margin threshold or no further under-grounded regions are detected. Because refinement operates at the segment level rather than paragraph level, inference cost is reduced by ~4× compared to VisVM (main manuscript Table 1).

Additional pseudocode for this procedure is provided in Algorithm 1 (lines 7–15), and the same beam width and temperature settings are used across both stages for fair comparison to baselines.

Table R1

Ablation results comparing ViMaR with default settings versus reduced diversity settings (T = 0.2, K = 2).

Dear Reviewer vUxp,
Many thanks to your professional, detailed, and valuable reviews. We address your concerns one by one below.

Question 1

We clarify that the reviewer’s concern arises from a misinterpretation of our formulation. In our framework, the reward in Eq. (1) is not purely “state-based” as implied. It is computed for the current state–action pair $(s_i, y_i)$, where $s_i = (y_i, I)$, with $y_i$ being the current sentence (including previously generated context) and $I$ being the input image. This reward $r_{s_i}$ is precomputed using the CLIP-based margin adjustment and does not involve any learning.

The training objective in Eq. (2) then uses this fixed reward to optimize the value function $V_ρ$ via temporal-difference learning, allowing $V_ρ$ to predict cumulative future rewards. This formulation is consistent with the standard MDP paradigm; our notation simply omits explicit action dependency for brevity.

Question 2

In our framework, the reward $r_{s_i}$ defined in Eq. (1) is precomputed using the CLIP similarity with margin adjustment and is not a learnable function through which gradients are propagated. The actual training objective is Eq. (2), where $r_{s_i}$ is treated as a fixed target and combined with the predicted value $V_ρ$ in a temporal-difference (TD) loss. Therefore, the reviewer’s concern regarding a discontinuous gradient $\frac{\partial r_{s_i}}{\partial \delta}$ at $\delta=\tau$ does not apply, as no gradient flows through Eq. (1) during optimization.

Regarding the design choice: the sharp margin is intentional. Our goal is to strongly penalize hallucinated captions (low $\delta$) and reward grounded ones (high $\delta$), making the separation between positive and negative cases explicit. This simple thresholding provides a clear and interpretable signal for the value model, and our ablations (Appendix D, lines 547–563) confirm robustness: small variations in $\tau$ do not destabilize training or degrade performance.

In summary, the margin-based reward is a fixed supervisory signal rather than a trainable component, and the reported results (Tables 1–2) empirically validate its stability and effectiveness for hallucination mitigation.

Question 3

We appreciate the reviewer’s suggestion to compare Eq. (1) with alternative formulations. We clarify our rationale and provide evidence as follows:

(a) $r_{s_i} = \delta$:
This formulation corresponds to the baseline CLIP-PRM approach and is effectively identical to the reward design explored in the VisVM paper. We also include this variant in our own baselines (Tables 1–2), where it underperforms our margin-based reward, particularly in hallucination mitigation, confirming that a naive use of $\delta$ is insufficient.

(b) $r_{s_i} = \max(\delta, \tau)$:
This design removes penalization for scores below $\tau$ by mapping all low-confidence sentences to the same value $\tau$, effectively treating mildly and strongly hallucinated sentences identically. This contradicts our objective of sharply distinguishing grounded and hallucinated captions. Empirically, this leads to weak gradients for low-quality candidates and fails to suppress hallucinations. Preliminary experiments confirmed inferior performance compared to our chosen design.

(c) $r_{s_i} = \delta$ if $\delta \ge \tau$, else $0$:
While this variant is superficially similar to our margin formulation, it removes negative penalization entirely and, critically, assigns the same reward to both mildly and strongly hallucinated sentences (all mapped to zero). This lack of differentiation weakens the model’s ability to prioritize corrections and leads to poorer sentence selection. Our experiments confirm this: performance drops noticeably and hallucination rates increase (results provided in Table R1), directly validating the necessity of our margin-based reward design.

Thus, our design explicitly penalizes under-grounded sentences rather than merely ignoring them, yielding significantly better alignment and stability, as supported by the quantitative results in Tables 1–2 and Table R1.

Question 4

The reviewer’s concern arises from a misinterpretation of our notation rather than the actual implementation. In our framework, the value function $V_ρ$ is computed over the entire state, which by definition (lines 133–136) includes the image $I$ and all previously generated sentences $y_{<i}$ (including $y_i$). Thus, when we write $V_ρ(y_{i+1}, I)$ in Eq. (2), this is a shorthand representation: $y_{i+1}$ is conditioned on the full prefix $y_{\leq i}$ and $I$, and the value function estimates the expected cumulative reward from this extended state.

In other words, our method does not discard contextual information from $y_i$; rather, it explicitly incorporates the full caption history up to step $i+1$ along with the image features. This design ensures proper state transitions and preserves long-range dependencies, which is essential for accurate paragraph-level captioning. We will clarify this explicitly in the final version.

Question 5

Our paper already describes the temperature strategy (Sec. 4.1): both BoN and ViMaR generate candidates using a fixed set of diverse temperatures {0.1, 0.3, 0.5, 0.7, 0.9} to promote diversity and capture both fine-grained and global details. This multi-temperature design is essential to our two-stage framework — Stage 1 benefits from broad candidate diversity, and Stage 2 selectively refines weak segments by leveraging candidates with varied entropy levels.

To directly address the reviewer’s suggestion, we performed additional experiments using fixed temperatures (e.g., T = 0.2) for both BoN and ViMaR. The results in Table R1 show consistent performance degradation: hallucination rates increase and caption richness declines compared to our multi-temperature setup. This is expected — fixed low temperatures (e.g., 0.2) lead to overly conservative, low-diversity captions missing fine details, while fixed higher temperatures (e.g., 0.6) increase diversity but may introduce hallucinations. Our multi-temperature scheme balances these extremes by mixing low-entropy and high-entropy samples, enabling the value model to select globally grounded yet detailed captions.

Thus, the proposed multi-temperature approach is not arbitrary: it is explicitly designed to enhance candidate diversity without sacrificing grounding, and both our main results (Tables 1–2) and new experiments confirm its superiority.

Table R1

ViMaR-guided decoding consistently enhances visual comprehension and mitigates hallucinations across multiple benchmarks, outperforming alternative strategies.

Table R2: ViMaR performance under different reward and temperature (T=0.6) strategies.

Thank you for the rebuttal.

W1: Based on the literal interpretation, there does not appear to be a misreading. In the current manuscript, the reward is denoted as $r_{s_i}$, which involves only $s_i$ and not $y_i$. Could the authors clarify whether they actually intended to use $r_{s_i,y_i}$ instead? If not, the current notation conflicts with the author's reply, which undermines the paper's clarity.

W2: This point is now much clearer. Thank you for the explanation.

W3: Points (a) and (c) have been addressed clearly. However, for point (b), I don't know why the authors only state that “Preliminary experiments confirmed inferior performance compared to our chosen design,” without providing the actual experimental results. Without such evidence, it is difficult to verify the claim of “inferior performance.” This is particularly valuable, as the authors mention in (c), that design “removes negative penalization entirely,” and (b) does not have this flaw.

W4: Once again, I believe this is literally not a misinterpretation. The notation is defined as $V_{\rho}(y_{i+1}, I)$ in the paper. Could the authors clarify whether they meant to use $V_{\rho}(y_{<i+1}, I)$ instead? Otherwise, this part of the manuscript again lacks clarity.

W5: The result presented in Table R1 provides a clearer justification for the effectiveness of dynamic temperature, which provides that using a fixed temp of 0.2 has no obvious gap compared to dynamic temperature. In addition, I don't know why the author didn't provide the result of temp=0.6. is it hard to run compared to 0.2? I would encourage the authors to include table R1 and provide more temperatures (e.g. 0.6) in the revised manuscript.

We truly appreciate your careful reading and constructive comments throughout the review process.

Thank you very much for your kind response and for taking the time to engage with our work so thoughtfully. We appreciate your constructive feedback. Please see below for detailed replies to your queries.

W1:

You are correct that in a standard MDP, the reward depends on both the state and the action. As mentioned in the main paper (Line 145) and also in our Rebuttal Q1, we include the action $y_i$ as part of the state definition: $s_i = (y_i, I)$

Using this definition, $r_{s_i}$ refers to the reward for a specific state–action pair at the current step. In other words, $r_{s_i}$ captures how well the current sentence $y_i$ aligns with the image $I$ based on the CLIP score.

W2:

Thank you for confirming. We are glad the explanation helped clarify this point.

W3:

Regarding point (b), we would like to clarify that the design $r_{s_i} = \max(\delta, \tau)$ also suffers from a fundamental issue: it removes the ability to differentiate between mildly and strongly hallucinated sentences, as already explained in our initial Rebuttal Q3 response. Specifically, if the reward score $\delta < \tau$, then $\max(\delta, \tau)$ will always assign the same fixed value $\tau$, effectively flattening the reward landscape below the margin. This eliminates the penalization gradient and treats all low-confidence outputs equally, regardless of their severity similar to point (c), which the reviewer acknowledges as problematic.

Although we were unable to include full experimental results for this variant in the initial rebuttal due to time constraints, we have since conducted the evaluation. As shown in Table R2 below, the performance of this design (denoted as ViMaR (Reward'')) is not only inferior to our proposed margin-based reward, but also underperforms the simpler reward variant tested in initial Rebuttal Table R1 (ViMaR (Reward')).

This validates our design choice and reinforces the importance of retaining a fine-grained penalization mechanism for low-quality candidates. We will include these new results and expanded discussion in the final version of the paper to further clarify the limitations of alternative reward definitions.

W4:

As described in lines 133–136 and 193–197, as well as in Algorithm 1 (lines 9–15) and Rebuttal Q4, the value function $V_\rho$ is computed over the full state, which includes the image $I$ and the complete caption context up to the current point; that is, all previously generated segments $y_{<i}$ along with the current segment $y_i$.

Thus, the notation $V_\rho(y_{i+1}, I)$ is a shorthand representation that assumes $y_{i+1}$ is decoded conditioned on the full prefix $y_{\le i}$ and the image $I$. We agree that explicitly writing $V_\rho(y_{\le i+1}, I)$ would better reflect this structure and will revise the manuscript accordingly to prevent confusion.

W5:

We appreciate the reviewer’s interest in further examining temperature settings. Due to the tight rebuttal schedule, we initially included only results for $T=0.2$ in Table R1. We have since conducted experiments at $T=0.6$; the results are shown in Table R2 below, and we will add further findings as soon as they become available.

As expected, $T=0.6$ increases diversity but leads to slightly higher hallucination and reduced grounding quality compared to our multi-temperature setup. These results suggest that relying on a single fixed temperature, especially a higher one, may not provide the best trade-off between detail and reliability, while our multi-temperature strategy offers a more balanced approach.

We will include Table R1 and the newly added results (Table R2) in the revised manuscript to strengthen this discussion.

Dear Reviewer wMfE,
Thank you for your valuable comments and questions, which have been very helpful in clarifying and improving the presentation of our work. We address your points below:

Question 1

Our work already demonstrates cross-model generalization beyond LLaVA. In addition to LLaVA-Next-Mistral-7B, we explicitly evaluate ViMaR on LLaVA-OneVision-Qwen2-7B (Table 2, lines 305–309), showing consistent improvements across all benchmarks, including hallucination metrics (CHAIR, MMHal).

To further strengthen this point, we have extended our evaluation to an additional open-source VLM, Qwen2.5-VL-3B. Results (provided in Table R1) similarly confirm that ViMaR delivers substantial gains in both fidelity and hallucination mitigation, underscoring its model-agnostic applicability.

Finally, we emphasize that ViMaR operates purely at inference time and does not require access to model weights (as also noted in lines 52–56, where we highlight its plug-and-play generalizability across VLM architectures). This property makes the method directly applicable even to closed-source APIs such as GPT‑4o. While such evaluations are technically feasible, they are prohibitively costly to perform at scale; therefore, we prioritized comprehensive open-source benchmarks for reproducibility and resource efficiency. We will clarify this practical consideration explicitly in the final version.

Table R1

Comparison between the original Qwen baseline and ViMaR-guided decoding.

Question 2

We thank the reviewer for raising this important point. We acknowledge that naive use of CLIP similarity can be limited in both precision and its ability to capture fine-grained, sentence-level details (as reflected in Tables~1,2 CLIP-PRM study). Our approach specifically addresses these concerns in three ways:

1. Temporal-Difference Value Model Beyond Raw CLIP Scores:
We do not rely on raw CLIP similarity alone. Instead, we use CLIP scores to calculate a Margin-Based Reward Adjustment within a temporal-difference (TD) value model (Sec. 3.1) that predicts long-term reward across the entire caption trajectory. This approach smooths local noise in individual scores by considering both the immediate alignment of a sentence and its downstream influence on subsequent sentences. Empirically, this yields more discriminative value estimates (Fig. 2), avoiding the instability noted with raw CLIP usage.

2. Margin-Based Reward Adjustment for Fine-Grained Segments:
To penalize under-grounded predictions (low δ), we introduce a margin-based penalty (Eq. 1) that calibrates the reward for partial sentences. Rather than treating all low scores equally, sentences below the calculated threshold τ (see Appendix D) are penalized proportionally according to their similarity scores. This mechanism effectively normalizes the CLIP reward score, allowing it to distinguish visually grounded details without over-penalizing neutral content.

3. Empirical Justification and Ablation Results:
We conducted extensive evaluations (Tables 1–2) demonstrating that our CLIP-based value model, when combined with margin calibration and two-stage refinement, consistently reduces hallucination rates (CHAIR_s from 26.2% to 23.1%) and improves factual richness compared to VisVM and other baselines. Importantly, this improvement holds even when applied cross-model (e.g., LLaVA-OneVision-Qwen2-7B or Qwen2.5-VL-3B) in the main manuscript Table 2 and Table R1, suggesting robustness beyond any single similarity metric.

In summary, while raw CLIP similarity alone is imperfect, our value-guided framework addresses its shortcomings via TD learning, margin calibration, and segment-level refinement, leading to demonstrably better grounding and efficiency over prior methods.

Question 3

This point is already addressed in our paper (Appendix D, lines 547–563). Specifically, we describe that the threshold τ was derived from the empirical CLIP similarity distribution on the training set (17th percentile), ensuring that only visually under-grounded sentences receive a penalty.

As further detailed in lines 147–151 (Equation 1), τ serves as the calibrated margin that determines when and how strongly low-confidence sentences are penalized: sentences above τ are rewarded proportionally to their CLIP alignment, while those below τ incur a penalty proportional to the gap. This mechanism is central to reducing hallucination without over-penalizing neutral or background details, and its role is explicitly described in the cited section. We further note that small variations in τ yield minimal performance change, indicating robustness.

Question 4

We already included some visual aids to clarify the method and results. These figures were designed to complement the algorithmic description in Section 3 (Algorithm 1) and provide both conceptual and qualitative clarity. We agree that adding a high-level schematic of the two-stage pipeline (coarse caption generation and targeted refinement) could further improve readability, and we plan to include such a simplified diagram in the camera-ready version for additional clarity.

Dear Reviewer 3B7b,
Thank you for your valuable comments and questions, which have been very helpful in clarifying and improving the presentation of our work. We address your points below:

Question 1

The effectiveness of the margin-based reward is already partially demonstrated in our paper through multiple quantitative and qualitative analyses. Specifically, Tables 1–2 and lines (254–267 and 289–309) report significant reductions in hallucination metrics (CHAIR$_s$, MMHal) and consistent improvements in visual comprehension benchmarks when the margin mechanism is introduced. Figures 2–5 further visualize how margin-penalized segments are selectively refined, yielding better grounding compared to other methods.

Isolating the margin’s contribution

1. Our analysis inherently isolates the contribution of the margin-based reward: the only modification is replacing the raw similarity signal with our margin-adjusted formulation combined with targeted refinement. This change alone yields a 3–4% reduction in hallucination rates and a 15.87% average improvement across benchmarks.

2. $r_{s_i} = \delta$: This formulation corresponds to the baseline CLIP-PRM approach and is effectively identical to the reward design explored in the VisVM paper. We also include this variant in our own baselines (Tables 1–2), where it underperforms our margin-based reward, particularly in hallucination mitigation, confirming that a naive use of $\delta$ is insufficient.

3. To further strengthen this point, we compared our margin-based reward against a simplified variant defined as:

r_{s_i} =
    δ, if δ ≥ τ
    0, otherwise

This variant removes the negative penalization component. As expected, it resulted in noticeably degraded performance and increased confusion during sentence selection. This outcome directly supports the necessity of our reward modeling design, and the quantitative evidence is provided in Table R1 (ViMaR(Reward′) entry).

Extended Results

To provide additional clarity, we have expanded our analysis beyond what was presented in the original submission:

1. Temperature control: We additionally analyzed ViMaR under fixed temperature decoding (e.g., T = 0.2) versus BoN and our multi-temperature setup. Results in Table R1 confirm that diverse temperature sampling is a key factor in ViMaR’s superior performance relative to Best-of-N baselines.

2. Cross-model evaluation: We tested ViMaR on the open-source Qwen2.5-VL-3B model in Table R1, which already exhibits strong baseline grounding. Despite this, ViMaR still provided measurable improvements in hallucination reduction and descriptive richness, confirming the generality of our approach.

Above analyses, combined with the results already in the paper, rigorously validate both our architecture and the margin-based reward formulation. They demonstrate that the margin not only improves hallucination mitigation but also enhances cross-model generalization and robustness across decoding strategies.

Table R1

ViMaR-guided decoding consistently enhances visual comprehension and mitigates hallucinations across multiple benchmarks, outperforming alternative strategies.

Question 2

We note the reviewer’s request for additional clarity regarding Stage 2 of our two-stage inference framework. While Stage 1 identifies the highest-value caption via best-of-N generation guided by the value model $V_ρ$, Stage 2 is specifically designed to refine only visually under-grounded segments rather than regenerate the entire caption, reducing compute overhead while improving fidelity.

1. Segment Identification. After Stage 1 generates the coarse caption $y^*$, we compute sentence-level value scores for each segment $y_i$ relative to the image. Segments falling below the calibrated margin threshold 1.87 (similarly calculated as τ) are flagged as “under-grounded” and removed from the coarse caption. Additionally, missed visual entities (e.g., objects detected in image features but absent from the caption) are flagged via cross-attention coverage analysis.

2. Candidate Generation. For each flagged segment, we condition decoding on the preceding accepted segments $y_{<i}$ and image $I$, sampling K = 6 candidates per temperature from the fixed set {0.1, 0.3, 0.5, 0.7, 0.9} (same as Stage 1). These candidates are short continuations targeted to replace only the flagged segment.

3. Scoring and Selection. Each candidate is scored using the value model $V_ρ(s,I)$, which is trained using temporal-difference value estimation (Sec. 3.1). The highest-value candidate replaces the flagged segment in the caption.

4. Iterative Refinement and Termination. This process iterates until all flagged segments exceed the margin threshold or no further under-grounded regions are detected. Because refinement operates at the segment level rather than paragraph level, inference cost is reduced by ~4× compared to VisVM (Table 1).

Additional pseudocode for this procedure is provided in Algorithm 1 (lines 7–15), and the same beam width and temperature settings are used across both stages for fair comparison to baselines.

Question 3

We thank the reviewer for pointing out this typo. Indeed, there was a typographical error in the “otherwise” branch of Eq. 1 in the draft. The intended reward formulation penalizes under-grounded predictions (low δ) rather than assigning them the same reward as well-grounded ones. Specifically, when δ < τ, the reward should be computed as:

r_{s_i} = δ − τ

which yields a negative value and therefore acts as a penalty within the temporal-difference objective. This ensures that low-confidence (under-grounded) sentences are explicitly down-weighted during value estimation, encouraging the model to learn a sharper separation between well-grounded and poorly grounded captions.

For clarity, the corrected Eq. 1 is:

r_{s_i} =
δ, if δ ≥ τ
δ − τ, if δ < τ

Here, δ ∈ [0, 1] denotes the CLIP similarity score, and τ = 0.16 is a calibrated margin derived from the empirical score distribution (see Appendix D). We will update the manuscript to reflect this corrected equation and prevent ambiguity.