DocVXQA: Context-Aware Visual Explanations for Document Question Answering

The authors sincerely appreciate Reviewer Y25T’s constructive feedback. We are pleased that the reviewer recognized the importance of explainability in Document VQA and the strong motivation behind our loss components. We also appreciate the positive remarks on the clarity and coherence of our writing and the value of our user study.

We have carefully addressed all the points raised in the following, and we are grateful for the detailed identification of typos and missing content, which will be corrected in the revised manuscript

1. "Self-explanatory" clarification: We clarify that our intention is not to suggest that we reveal the DocVQA model internal workings. Rather, we use "self-explanatory" following established usage in explainable AI literature (Mersha et al., Neurocomputing'24), (Gautam, Boubekki et al., NeurIPS'22), describing models inherently designed to produce interpretable outputs during inference (L158–162, left). This contrasts with post-hoc methods, which retrospectively assess relevance between inputs and outputs on frozen models. We fully accept the reviewer’s recommendation and will clearly specify that self-explanatory refers exclusively to "the model’s capacity to produce interpretable outputs by design," not exposing internal workings.

2. Model-agnostic claim is not supported: As we also commented in response to reviewer ovki, our claim of model-agnostic nature relies on the independent, modular design of the explanation mechanism. Although currently integrated with Pix2Struct, its components, such as a learnable mask head, three loss terms of the explainability principles, and a postprocessing pipeline, are architecture independent. To empirically validate this, we are currently integrating our approach with Donut (Kim et al., ECCV'22), and results will be provided in the final manuscript.

3. The encoder was pretrained only on original documents, the reviewer suspects that a domain gap occurs when it is applied on the masked ones. Using S+M, the answer is generally visible, it should be easy to produce the correct one, even without ColPali prior: The reviewer points out low utility scores in the sufficiency and minimality ablation (S+M) despite visible answers in masked documents. We respectfully disagree. Although answers are sometimes visible, this is not consistently true. The domain gap noted by the reviewer is also reduced with finetuning on original and masked inputs. We identify aggressive masking as the primary factor lowering utility scores, enforcing minimality at the cost of sufficiency. This lack of context impairs predictions, highlighting the importance of integrating the ColPali prior through token interactions (Table 2).

4. The paper does not explain why the decoder is kept frozen: We freeze the decoder to preserve its pretrained text generation capabilities and maintain stable training outputs. Since the decoder is pretrained on original, unmasked documents, jointly finetuning it with the encoder and mask head introduces distributional shifts, causing instability. Instead, we focus learning efforts on the encoder and the new mask head. Noting that we did tried unfreezing the decoder, but it led to unstable optimization, slower convergence, and reduced overall performance. We will add these details to the final manuscript.

5. Using ColPali to have more context-aware explanations will increase pixel ratio without increasing utility.: Table 2 shows that using only sufficiency and minimality losses (S+M) yields accuracy of 0.19 and ANLS of 0.36. Integrating ColPali through token interactions (S+M+TI) significantly improves these metrics to accuracy 0.38 and ANLS 0.54. Although ColPali increases the pixel ratio, this reflects the preservation of essential contextual cues rather than irrelevant areas. Practically, this additional context is critical for robust predictions, accuracy and mask learning guidance. Qualitative human evaluations further confirm that ColPali-enhanced explanations are clearer and more context-aware, supporting a balanced trade-off between minimality and sufficiency.

6. The proposed model yields results that are very close to the ColPali + Pix2Struct baseline: Figure 3 illustrates that relying solely on quantitative metrics is insufficient for comprehensive evaluation. Although quantitatively our results appear similar to ColPali+Pix2Struct, qualitatively ours perform better. Following the reviewer’s suggestion, we conducted a human preference study comparing masks from both methods. In total, 12 participants evaluated 21 randomly selected question-answer pairs (252 trials). Our method was preferred in 163 trials (64.7%; 95% CI [58.4%, 70.6%], p << 0.001), with all participants (12/12) favoring our approach overall. Thus, our method is close to ColPali+Pix2Struct quantitatively, but it is offering a significantly more compact and interpretable explanations. This new study will be added to the revised manuscript.

The authors thank Reviewer 9NT1 for their feedback. We appreciate the recognition of our approach’s novelty in addressing the challenge of transparency in DocVQA systems, and the acknowledgment that our approach provides a strong theoretical foundation through the quantitative formulation of explainability principles. Below, we address all of 9NT1’s concerns.

1. What is the performance on documents with complex layouts or multiple tones of writing? What are the limitations in handling highly complex or densely populated docs?: In the first case, we found that our model still performs well and produces good explanations. In densely populated documents, our model can provide fragmented explanations, which are hard to interpret. See qualitative results here: https://shorturl.at/nstj0. We will discuss these results in the final manuscript.

2. What are the computational costs compared to baseline approaches?: Our method requires additional training compared to post-hoc approaches. However, at inference time, the additional computational cost is only the forward pass through the small mask proposal head (591361 parameters). All baseline comparisons are performed with respect to the Pix2Struct base model with 282M parameters, aside from ColPali, which itself uses 2,9B parameters.

3. Can the method be extended to other vision-language understanding tasks? Why using of different metrics than ColPali and Pix2struct?: The masking-based method can be readily generalized to other vision-language tasks. Nevertheless, the primary goal of our work is to provide explanations tailored for the DocVQA task, rather than aiming at making Pix2Struct itself explainable. For the metrics used, we clarify that our specific focus is on DocVQA, thus retrieval metrics such as NDCG (used in ColPali) are not applicable. We adopt standard DocVQA metrics that are also used by Pix2Struct for QA tasks.

4. What is the relation of your work with mutual information estimation?: We appreciate the reviewer’s reference to Doshi-Velez and Kim (2017). Our core motivation closely corresponds with both the data-driven methodology for uncovering interpretability factors (Section 4.1) and the functionally-grounded evaluation outlined in Section 3.3. We concur with the suggestion of adopting an application-grounded evaluation framework; we plan to apply our methodology to domain-specific DocVQA tasks in future research.

5. How robust is the method to variations in document image quality and why there is a degradation of performance on masked images?: SPDocVQA contains some images of bad quality, thus our model is robust to these variations, some qualitative results here: https://shorturl.at/mqbzg. Regarding the performance degradation from masked to unmasked models: the trade-off between accuracy and explainability is a well-documented phenomenon (Crook et al., REW'23, Bell et al., ICPS'22), and several approaches have been proposed to balance this trade-off (Luo et al., BJR Open'19, Mori et al., ESE'23). As illustrated in Fig. 4: lowering the threshold decreases explanation quality but improves utility. This shows that the observed performance gap is an inherent constraint in explainability rather than a fundamental limitation of our approach. We also want to clarify also that the 38% is the accuracy of masked images; for clear images, the accuracy is 51%.

6. Why using different thresholds and why there is a heavy reliance on post-processing: We adjust the thresholds so that each technique operates at its optimal setting, to ensure a fair comparison between competing methods. The post-processing steps are primarily intended to enhance human interpretability of the masks, and do not significantly affect the core performance of our model. As shown in Figure 4, the observed gains are primarily attributable to the underlying learning approach rather than to the post-processing.

7. The paper lacks a proper ablation study showing how ColPali and Pix2Struct perform individually: We would like to clarify that it is impossible to evaluate each technique's individual impact, as they are designed to be used only in combination for XDocVQA.

8. The baseline numbers in the top row appear to outperform many of the tested approaches. We would like to clarify that these are not the baselines but rather upper bounds (unmasked, no explanation). We will make this distinction clear in the paper.

9. Human evaluation lacks statistical rigor with 26 participants evaluating only 10 examples: It is challenging to include more documents because we have to redo the study, but we have increased the number of participants to 42, and found that the results remained consistent as seen in this table:

The authors sincerely appreciate Reviewer ovki’s insightful comments. We are pleased that the reviewer recognized the paper’s key strengths, as he mentioned its originality in integrating interpretability into DocVQA, its effective balance between interpretability and prediction performance, and the "conceptual soundness" of our theoretical formulations. We have carefully considered all suggestions and made a dedicated effort to address each point thoroughly.

1. I cannot confirm the generalizability of your approach. Additional experiments on diverse datasets are needed: We have extended our evaluation beyond the standard DocVQA benchmark SPDocVQA (Mathew et al., WACV'21) by incorporating two additional datasets: InfographicVQA (Mathew et al., WACV'22) and PFL-DocVQA (Tito et al., ICDAR'24). These datasets pose greater challenges due to their complex and heterogeneous visual-textual structures, providing a more rigorous test of our model’s generalization capabilities. Some qualitative results on InfographicVQA are available here: https://anonymous.4open.science/r/RebuttalICML2-4D0C/README.md. The new results are demonstrating our model’s ability to handle these visually complex documents. As seen in the examples, our approach yields promising results, indicating strong potential for generalization. We will provide the full results on these new datasets in the final manuscript.

2. The proposed approach was evaluated using only one model (Pix2Struct), so the claim of being "model-agnostic" is not supported: As we also commented in response to reviewer Y52T, our claim of model-agnostic nature relies on the independent, modular design of the explanation mechanism. Although currently integrated with Pix2Struct, its components, such as a learnable mask head, three loss terms of the explainability principles, and a postprocessing pipeline, are architecture independent. To empirically validate this, we are currently integrating our approach with Donut (Kim et al., ECCV'22), and results will be provided in the the final version of the paper

3. The performance drop (56%→51%) introduced by interpretability constraints is not sufficiently explained or analyzed.: It can be attributed to the additional double-loop process employed by our approach. Even when the mask $M$ is set to all ones so that $X \odot M = X$, the model still performs two passes through the encoder—one for mask generation and another for final prediction. Consequently, during backpropagation, gradients from both passes are accumulated. In theory, if both passes were identical, this would be equivalent to a single pass. However, in practice, non-linearities, normalization layers, and other factors introduce slight variations between the two passes. These differences can cause over-accumulation or interference in the gradients, perturbing the weight updates relative to the original training regime. Such deviations, along with potential numerical instabilities from the extra pass, likely contribute to the modest performance drop observed from 56% to 51%.

4. The choice of a fixed parameter (top-$k$=3 regions) in mask selection lacks experimental justification. Providing sensitivity analysis for different values of $k$ is recommended: Following the reviewer’s suggestion, we conducted a sensitivity analysis on the hyperparameter $k$ that determines the number of regions selected during mask postprocessing. The results are summarized in the table below. As the table indicates, increasing $k$ leads to gradual improvements in both accuracy and ANLS, but it also results in a higher pixel ratio, which reflects a less minimal explanation. In terms of balancing utility and minimality, our experiments suggest that choosing $k=3$ or $k=4$ offers a good trade-off. That is, these values yield utility metrics comparable to higher $k$ settings while keeping the highlighted area compact. We will include the complete sensitivity analysis table in the Appendix of the revised manuscript.