CF-VLM：CounterFactual Vision-Language Fine-tuning

Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.

L1. The loss term L_{csd} does not make sense.

A1: Thank you for raising concerns about $L_{csd}$. This loss is not intended to punish logically coherent counterfactuals. Instead, its margin‑hinge form (Eq. 2) activates only when the embedding of a counterfactual scene $(I_{cf_k},T_{cf_k})$ encroaches too closely on the anchor $(I_a,T_a)$, thereby preserving a unique and stable semantic cluster for factual reality and preventing multiple “parallel worlds” from collapsing into one representation. Complementing this scene‑level separation, $L_{fcd}$ targets fine‑grained attribute edits. Thus, we keep their weights comparable in the overall objective $L=\alpha L_{align}+\beta L_{csd}+\gamma L_{fcd}$ with $\beta\approx\gamma$ to balance learning across semantic levels. Empirically, switching from our default SDXL‑generated counterfactuals (ConMe 58.9 %, ARO 88.5 %, CLIP 0.92, KL 0.10) to a fully real‑image set (RealCF: ConMe 59.2 %, ARO 89.0 %, CLIP 0.95, KL 0.05) yields essentially identical performance, demonstrating that $L_{csd}$ enforces semantic separation rather than detecting synthetic artefacts. A sweep varying $\beta$ (and $\gamma$) from 0.30 to 0.60 changes ConMe and ARO by less than one percentage point, confirming robustness. In the revised paper, we have added an ablation with $\beta=0$, provide full RealCF tables including lower‑$\beta$ settings, and introduced a curriculum that linearly decays $\beta$ as counterfactual quality (measured by CLIP similarity) approaches that of the anchor. In summary, these results highlight $L_{csd}$ as a key component that, alongside $L_{fcd}$, enables the model to discriminate semantic differences at multiple levels.

---

P1&P2&P3. Paper Formatting Concerns

A2: Thank you very much for your valuable comments. We have incorporated the corrections into the revised paper to ensure rigor and professionalism in our presentation. We acknowledge the issue with inconsistent LaTeX notations for symbols such as $I_{cf_k}$, $I_{cf_{edit_j}}$, and $I_{cf_{edit}}$. We have conducted a thorough proofread and unified all related variables into a strict and consistent format. For instance, $(I_{cf_k}, T_{cf_k})$ from Line 145 and $I_{cf_{edit_j}}$ from Line 147 are now standardized as $(I_{cf_k}, T_{cf_k})$ and $I_{cf_{edit_j}}$. Specifically, in Section 3.2 and Equation (3), we have corrected all inconsistent notations like $S(I_{cf_{edit_j}}, T_a)$ to $S(I_{cf_{edit_j}}, T_a)$ and have ensured this standard is applied consistently throughout the paper. Thank you once again for your correction. It is crucial for enhancing the quality of our paper.

Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.

W1: The writing quality is not high enough; there are typos in both the text and figures, which affects reading comprehension to some extent.

A1: Thank you for pointing out these. In the revised paper, we have conducted a thorough proofreading and polishing of the entire paper, fixing all identified spelling, grammar, and formatting errors, and optimizing some sentences for clarity. For example, we have corrected the missing space between 'Models: The is' in Line 84 to ensure proper formatting. We believe these revisions have significantly improved the overall quality and readability of the paper. Thank you again for your valuable comments.

---

W2: I think the paper's explanation of how counterfactual data is constructed and how to ensure the modifications are genuinely causal is not thorough enough.

A2: Regarding your concerns about counterfactual construction and causality assurance, we have provided detailed clarifications in Section 2.2 and the appendix of the revised paper. The core of ensuring our modifications are causal lies in the Minimal Intervention principle. We only change one independent semantic variable at a time (such as a single attribute or a causal relationship) while keeping the rest of the scene unchanged. This allows us to attribute any change in the model's performance to this single modification. Our CF-VLM uses carefully designed prompts to guide an LLM (Qwen2-72B-Instruct) to make minimal edits to the text, and then a text-to-image model (SDXL) generates the corresponding counterfactual image. For example, for the fact 'a man in a blue shirt kicks a ball,' an attribute counterfactual would be 'a man in a red shirt kicks a ball,' while a causal counterfactual would be 'a man misses the ball.' This structured intervention ensures the specificity and causal validity of the counterfactual samples. We believe these clarifications more thoroughly explain the rigor of our method.

---

Q1: The variable subscripts in lines 144 to 147 are quite messy, which I found confusing. Could the authors explain it further? Additionally, should the Fine-Grained Causal Discrimination Loss in line 184 be placed on a new line?

A1: Sorry for the confusion. In the revised paper, we have added the following supplementary explanations in Section 3.1:

• Anchor factual pair $(I_a, T_a)$: The original, unmodified image-text pair, serving as the baseline sample for the model.
• Complete counterfactual scenario pairs $\{(I_{cf_k}, T_{cf_k})\}_{k=1}^K$: Both the image and text are modified simultaneously to construct a new, logically self-consistent scenario that is different from the factual one.
• Minimally edited counterfactual images $\{I_{cf\_edit_j}\}_{j=1}^J$: A single key semantic edit is applied to the image, which is then paired with the original text $T_a$ to train the model's sensitivity to fine-grained changes.

Furthermore, regarding the description of the loss function in Lines 183-187, we completely agree with your assessment. In the revised paper, we have changed this section to a list format, presented under the subheading "Fine-Grained Causal Discrimination Loss," to enhance the paper's structure and readability.

---

Q2: In Figure 2, the L_align subpart, the subscripts are incorrect-they should be T_cf1 to T_cf4. In Figure 2, the original image referenced in the rectangle to the right of the counterfactual loss is incorrect.

A2: We have conducted a thorough proofreading and revision of our paper based on your suggestions. Here is our point-by-point response:

In the original figure, the intermediate step correctly labels the image of 'walking in the park' as 'The hardest' negative sample. However, in the final calculation of the fine-grained causal discrimination loss ($L_{fcd}$) on the right, we inadvertently used the image with the 'beach' background, causing a logical inconsistency in the diagram. In the revised paper, we have corrected this by replacing the 'beach' image in the loss calculation box on the right with the image of 'running in the park,' ensuring the diagram for the entire $L_{fcd}$ calculation process is rigorous, accurate, and logically consistent.

Furthermore, regarding the subscript notation $T_{cfk}$ in the $L_{\text{align}}$ equation, our original intention is to use the index $k$ to indicate that the original image would be aligned with the text corresponding to each of the counterfactual images. However, as you pointed out, this notation could indeed be ambiguous in the current context. We are very grateful for this reminder and have modified the relevant notation according to your suggestion to improve the clarity and readability of the formula.

To prevent similar confusion for other readers, we have significantly expanded the caption for Figure 2 in the revised paper, providing detailed textual explanations for both of these points.

---

Q3: How do you ensure that the modification to the image is minimal? How do you determine that the modified part corresponds to a causal attribute, or that the modification reflects changing a causal relationship? I didn't find Section 2.3 as you mentioned in line 189.

A3: To ensure that image modifications are both minimal and reflect changes in causal mechanisms, we employ a systematic strategy. First, we use the multi-modal large model Qwen2.5-VL to identify the minimal visual anchors corresponding to the text description. We then strictly constrain the editing operation to within these anchor regions, specifying the object and attribute to be modified via masks or control instructions (e.g., changing only the motion state of the ball), thus ensuring the locality and target-consistency of the modification. Building on this, to ensure these minimal modifications truly reflect causality, we further extract the causal chain structure from the image and text (e.g., "kick ball → ball flies"). When generating counterfactual samples, we explicitly intervene on the antecedent of the causal chain (the action 'kick') to deduce a logically sound consequent ('ball is stationary'), which is equivalent to simulating a 'causal break' in the visual space. Finally, we use joint judgment from both language and vision models to verify the logical consistency and semantic contrast of each counterfactual sample. Only when the modification creates a clear semantic negation and is consistently recognized as plausible by the models do we consider it causally valid.

Regarding Section 2.3 mentioned in Line 189, we sincerely apologize for the typos. We mistakenly direct you to Section 2.3. The detailed explanation of how we define and classify counterfactual samples (such as single attribute modification and key causal relationship adjustment) is correctly located in Section 2.2 (Definition and Types of Counterfactual Samples). We have carefully checked and corrected all cross-references in the final revised version for accuracy. Thank you very much for pointing this out.

Dear ZY2y ,

Thank you for your thoughtful follow-up and for carefully reviewing our response. We greatly appreciate your decision to raise the score to 4 and recommend acceptance. Your support is truly encouraging.

We have taken your feedback regarding typos and writing issues in the original submission seriously. In the revised paper, we have made significant efforts to enhance the writing quality, ensuring greater clarity and professionalism throughout. Additionally, within the next two days, we will provide detailed revision comments in the Official Comment section.

Once again, we sincerely value your constructive feedback and support throughout the review process.

Best regards, The Authors

Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.

W1: CF-VLM seems to be a very fine-grained version of TripletCLIP, with more refined semantics, more refined negative samples, and more detailed objectives that address different aspects of vision-language understanding.

A1: We thank the reviewer for pointing out the connection between CF-VLM and TripletCLIP within the contrastive learning framework. Indeed, both formally use a triplet structure, but our CF-VLM differs fundamentally in its objectives, supervision signal design, and task setting:

• Objective Difference: TripletCLIP primarily aims to enhance the discriminative representation capabilities of VLMs. In contrast, CF-VLM, while strengthening fine-grained semantic understanding, further focuses on improving causal reasoning capabilities to overcome the common bottleneck VLMs face in complex reasoning tasks.
• Negative Sample Design: TripletCLIP typically uses random or semantically adjacent negative samples and guides the model to focus on details by simultaneously transforming multiple attributes. However, this can lead to generated samples having an overly large semantic gap from the original image, diminishing the effect of minimal changes. In contrast, CF-VLM uses structured interventions to generate negative samples with clear counterfactual semantics and minimal critical differences. This more effectively enhances the model's sensitivity to minimal attribute changes and, by introducing causal counterfactuals, further boosts its causal reasoning abilities.
• Richer Hierarchy of Training Objectives: CF-VLM designs three complementary training objectives ($L_{align}$, $L_{csd}$, and $L_{fcd}$) that constrain the model's behavior from three dimensions: cross-modal alignment, counterfactual discrimination, and causal sensitivity. This far exceeds the expressive capacity of typical triplet-based contrastive learning.

Therefore, while CF-VLM borrows from the contrastive learning paradigm, we believe it represents a substantial extension in task setting, objective design, and supervision granularity, aimed at addressing the key bottlenecks of current VLMs in fine-grained understanding and causal modeling. We have further emphasized the differences and positioning between our CF-VLM and TripletCLIP in the revised paper.

Q1: What is the performance of composite changes (i.e., combining all the separate changes of a sample together and fine-tuning)?

A2: To investigate the effect of composite modification (combining all individual changes of a sample) in fine-tuning, we design a comparative experiment to directly compare structured counterfactuals (intervening on only one semantic attribute at a time) with composite counterfactuals (modifying multiple attributes simultaneously).

Based on the table results, we can see that the performance of CF-VLM (composite) is significantly lower than that of the standard CF-VLM. Specifically, the Conme and ARO scores on ViT-B/32 and Qwen-VL 7B are lower, indicating that the composite change strategy does not effectively improve model performance.

The composite change strategy is effectively the same as the multi-attribute joint modification method in TripletCLIP. As we have emphasized, guiding the model to focus on details by simultaneously transforming multiple attributes can lead to an overly large semantic gap between the generated sample and the original image, diminishing the effect of minimal changes. In contrast, CF-VLM uses structured interventions to generate negative samples with clear counterfactual semantics and minimal critical differences. This more effectively enhances the model's sensitivity to minimal attribute changes and, by introducing causal counterfactuals, further boosts its causal reasoning abilities.

Q2: Does the scale of fine-tuning data affect performance?

A3: We have already explored this through detailed ablation studies in our original paper, with the relevant analysis presented mainly in Section 4.3 (right side of Figure 4), Appendix E (Figure 9), and Appendix F. As we stated in these sections, our CF-VLM's performance does indeed improve with an increasing amount of counterfactual data; specifically, experimental results show that on the ConMe benchmark, our CF-VLM's average accuracy steadily increases as the proportion of counterfactual samples grows. However, this analysis also reveals a phenomenon of diminishing marginal returns, with the performance curve starting to plateau after the ratio reaches 60%-80%. To quantify this effect more precisely, we conduct deeper ablation studies in Appendix E and Appendix F. The results show that as the ratio of counterfactual to factual samples (K) increases from 1 to 4, performance on all tasks improves monotonically, peaking at K=4. When the value of K continues to increase beyond 4, the performance gains stagnate or even slightly decline, which may be due to over-regularization caused by excessive exposure to perturbations. These findings collectively reveal a practical trade-off between performance improvement and computational cost. Therefore, based on this evidence, we empirically recommend that K=4 is a cost-effective configuration.

Q3: Compared to TripletCLIP, does the advantage of CF-VLM also lie in its need for fewer samples to achieve similar performance?

A4: Yes. To systematically validate this hypothesis, we set performance thresholds on three evaluation benchmarks: Conme (Avg) = 50, ARO (Avg) = 75, and VL-Checklist (Avg) = 75. During the training process for both TripletCLIP and CF-VLM, we evaluate every 100k samples processed. We observe the amount of data required for each model to first reach its respective performance threshold, thereby comparing their differences in training efficiency and sample utilization.

We compare the sample efficiency of TripletCLIP and CF-VLM at fixed performance thresholds. As shown in the table, CF-VLM reached the target on Conme, ARO, and VL-Checklist with 9.6M, 11.2M, and 10.8M training samples, respectively, significantly outperforming TripletCLIP's 11.3M, 13.6M, and 12.6M. We believe this advantage stems from the fact that counterfactual images not only form effective contrasts with the original images but also help construct multi-dimensional semantic contrastive signals among the counterfactual samples themselves. This significantly reduces the model's dependence on the volume of training data while ensuring performance.

Q4&Q5_A: I found some typos: In line 309, the topic is missing. In line 84, a space is needed between 'Models: The is'.

A5: Thank you very much for the corrections; we have updated them in the revised paper. We have added the topic sentence "Inspired by these works, our research is the first to extend counterfactual reasoning from the field of model explanation to the model fine-tuning stage to enhance model performance" in Line 309, and we have also corrected the spacing issue in Line 84.

W1. What are the original contributions?

A1: Our CF-VLM effectively mitigates the limitations of VLMs in fine-grained understanding and significantly enhances their causal reasoning capabilities by introducing minimally edited counterfactual samples. The core of CF-VLM is our designed structured minimal intervention strategy for generating counterfactual samples, which creates controllable training data through minimal attribute-level modifications and causal relationship reconstructions. We further integrate three complementary training objectives, maintaining cross-modal alignment ($L_{align}$), discriminating scene-level counterfactuals ($L_{csd}$), and perceiving minimal causal perturbations ($L_{fcd}$), to construct a unified counterfactual supervision signal. Extensive experiments demonstrate that our CF-VLM substantially outperforms existing SOTA methods on multiple fine-grained and causal understanding benchmarks, and exhibits stronger robustness and factual consistency on visual hallucination evaluation benchmarks like POPE and MME. To support community research, we have also constructed a dedicated counterfactual dataset containing approximately 2 million high-quality synthetic images. Therefore, our contribution lies not only in proposing a new framework but also in providing a solid foundation for VLMs in real-world scenarios that require high-level reasoning and interpretability by enhancing causal sensitivity.

W2: Core details are missing

A2: We have added a dedicated "Experimental Setup" subsection in Section 4 (Experiments) of the revised paper to centralize core experimental information. This section now concisely introduces the main datasets used for fine-tuning, such as CC12M (8.6M pairs) and CC3M (2.6M pairs), and futher outlines the counterfactual sample generation process, including the text generation model (Qwen2-72B-Instruct) and image generation model (SDXL) used, as well as specific editing strategies (e.g., attribute, object, and relation modification); and specifies key hyperparameters like batch size (256), optimizer (AdamW), and learning rate (1×10⁻⁵). Furthermore, we clarify the 1:1 ratio of factual to counterfactual samples used in the main experiments, while also noting that other ratios, including 1:4, are explored in ablation studies (see Figure 4 and Appendix F) to eliminate potential ambiguity.

We have also reorganized and optimized Appendix A (Dataset Details) and Appendix C (Detailed Experimental Setup) in the revised paper. These appendices now serve as a clearer and more comprehensive reference, including full hyperparameter tables, detailed data split schemes, and the counterfactual rewriting prompts used for data generation. We believe these modifications significantly enhance the clarity and reproducibility of the paper.

W3: The limitations need more discussion.

A3: In complex or rare semantic intervention scenarios (e.g., subtle action changes or attribute combinations), SDXL may generate blurry images or images that do not conform to the desired intervention, leading to inconsistencies between visual content and the corresponding text. As presented in Appendix D, we use spatial masking and cross-attention control to apply local constraints to the intervention area, thereby improving the accuracy of local editing. We design a one-to-one attribute intervention mechanism, changing only one semantic attribute in the image at a time to minimize ambiguity and enhance generation stability. During the training phase, we have incorporated a cross-modal alignment objective ($L_{align}$) to further mitigate negative transfer caused by image-text errors, in addition to the counterfactual supervision loss.

**W4: How is counterfactual defined? **

A4: We define a counterfactual sample as a sample generated by applying a minimal intervention to a single semantic factor (e.g., color, pose, quantity, emotion) in an image or text, while keeping the main semantic context unchanged. A counterfactual does not necessarily involve a complete rewriting of the causal chain. It also includes fine-grained semantic adjustments, as long as they induce a contrastive change in the semantic state.

Specifically, a counterfactual sample meets two core conditions:

• Minimal Intervention: Compared to the anchor sample, there is a change in only one key semantic attribute, ensuring high contrast.
• Semantic Plausibility: The modified sample still forms a reasonable image-text pair without introducing obvious logical conflicts or fabricated scenarios.

Therefore, our data includes both counterfactuals with rewritten causal relationships (e.g., changing "causal direction" or "action outcome") and those with only minimal semantic perturbations (e.g., changing "red shirt" to "blue shirt"), which together form the core of the structured counterfactual supervision in our framework.

To ensure the modifications concentrate on core semantics, we input the image and corresponding text into Qwen2.5-VL, guiding the model to identify minimal semantic units and their corresponding visual anchors. After extracting key elements, we perform targeted minimal modifications to generate high-quality counterfactuals with clear semantic contrast. For visual counterfactuals, we further use image-text information to identify key causal chains (e.g., "kick ball → ball flies") and generate logically plausible counterfactual causal chains by modifying the causal condition (e.g., "does not kick ball" → "ball stays still").

Q1: Some comparisons and related works are missing.

A5: We have expanded the literature review in Section 5 (Related Works) of the revised paper. Specifically, we clarify that unlike VisMin [s1], which treats understanding minimal visual changes as the end goal of an evaluation task, CF-VLM uses such changes as a training mechanism to enhance the model's causal reasoning capabilities. Similarly, unlike AutoHallusion [s2], which focuses on automatically generating an evaluation benchmark to assess model hallucination, we utilize counterfactual samples as a core training signal to actively mitigate hallucination, rather than just for evaluation. Furthermore, we have also expanded our discussion of broader related work to further highlight the uniqueness of our framework. We believe these revisions provide a clearer positioning for our paper.

[s1] R. Awal et al., VisMin: visual minimal-change understanding, in NeurIPS, 2025.

[s2] X. Wu et al., AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models, in Findings of EMNLP, 2024.

Q2: The hallucination problem?

A6: The core of our CF-VLM lies in introducing a structured counterfactual supervision signal, which guides the model to focus on key semantic changes and can effectively mitigate hallucinations caused by "subtle causal inconsistencies between image and text" in VLMs. Firstly, CF-VLM employs a finely controlled counterfactual intervention mechanism to construct image-text pairs by making minimal modifications to images at the attribute or causal level, guiding the model to learn to identify subtle yet decisive semantic differences. Secondly, CF-VLM introduces structured semantic consistency constraints, combining cross-attention control and local spatial masking to improve the consistency between the visual response in the intervention area and its textual description. Overall, the mechanism of CF-VLM significantly enhances the model's stability and reliability under complex semantic interventions.

For more challenging hallucination benchmarks, we have integrated the table below.

As shown, CF-VLM significantly outperforms existing SOTA methods, indicating stronger factual consistency and resistance to interference. Particularly on MMHal, CF-VLM achieves a nearly 6-point improvement over standard fine-tuning, demonstrating its effectiveness in suppressing image-text hallucinations in highly adversarial scenarios, thanks to CF-VLM's minimal intervention counterfactual training mechanism.

**Q3: How is the generalization performance across datasets? Did you fine-tune on each dataset? **

A3: As described in the experimental setup, the pre-training and fine-tuning are conducted on the filtered CC12M (approx. 8.6 million image-text pairs) and its subset CC3M (approx. 2.6 million pairs), respectively. Additionally, we introduce approximately 40 million contrastive samples, co-synthesized by an LLM and a text-to-image (T2I) model, to enhance the counterfactual supervision capability. It is important to emphasize that to ensure fairness in the amount of training data across methods, not all data was used for fine-tuning. Instead, we uniformly control the number of training samples used for each method to be approximately 15 million pairs. When the primary data is insufficient, MSCOCO Captions are used as a supplement to maintain a consistent sample size.

Although our paper does not explicitly conduct "cross-dataset transfer" experiments, all models are trained on CC12M or CC3M, while evaluations have covered a range of tasks, including compositional and attribute-relation reasoning on benchmarks like Conme, ARO, and VL-Checklist. The generalization ability is still indirectly demonstrated through testing on multiple heterogeneous benchmarks, including zero-shot image classification on ImageNet-1k and image-text retrieval on MSCOCO and Flickr30k. This setup effectively justifies the CF-VLM's robustness in cross-task and cross-distribution scenarios.

Dear Reviewer LF2i,

Thank you for your time and valuable feedback, including your final justification. If our recent comments need further clarification or if you have additional thoughts, we’d be delighted to address them during the discussion period. Please feel free to reach out, and we’ll respond promptly with any needed information.

Warm regards,

Authors