Gradient Inversion of Multimodal Models

We thank the reviewer for his time, effort, and valuable suggestions.

Q1. "Tab. 2 cumulative.." + Q3."..channel and pixel-level TV.."

A1. We followed the standard practice of prior works such as GradViT (Hatamizadeh et al., 2022) and GradInversion (Yin et al., 2021), which apply priors incrementally and in combination rather than in isolation. These methods recognize that priors are often interdependent—for example, Total Variation enforces smoothness, while Laplacian sharpens edges, and together they produce better reconstructions than either alone.

Our goal in Section 4.2.2 was to show how combinations of priors improve reconstruction quality, particularly in multi-modal DQA settings, where each prior targets different aspects (e.g., edges, smoothness, spatial layout).

That said, we agree that isolating each prior provides additional insight. We have therefore conducted an ablation study evaluating each prior individually alongside the core gradient loss ($L_{grad}$).

The results show that while each prior contributes modestly on its own, the combination (presented in the paper) consistently outperforms any single prior, confirming the complementary nature of these losses. For the TV variants, we can see they contribute meaningfully on their own, but their combination consistently yields the best performance across all metrics. Note that these results differ slightly from those in the paper, as they were conducted on a data subset due to rebuttal time constraints. However, they clearly convey the main trend, and full results will be included in the final version.

Q2. "gaussian essential.."

A2. We would like to clarify that the absolute values of the loss weights do not reflect relative importance, as each term operates on a different scale. For instance, the Gaussian prior yields smaller values than the text loss, so its weight is lower to ensure balanced gradient contributions. Therefore, a smaller weight for the Gaussian prior does not imply that it is negligible—rather, it ensures balanced gradient contributions during optimization across all terms. This normalization prevents any term from dominating due to scale differences rather than actual relevance.

Q4. "selection of priors.."

A4. Our priors were selected for their ability to capture complementary and theoretically grounded aspects of the reconstruction process:

• Laplacian emphasizes high-frequency components, enhancing edge sharpness and fine details—critical for recovering small text.
• Gaussian applies a low-pass filter to promote global smoothness, stabilizing early optimization and reducing noisy minima.
• Total Variation (TV) encourages piecewise smoothness while preserving edges:
  • Spatial TV regularizes differences between neighboring pixels, reducing noise while preserving layout structure.
  • Channel-wise TV promotes consistency across color channels, helping suppress chromatic artifacts and preserve clean text appearance. Together, these priors form a multi-scale regularization strategy, guiding optimization from low-level structure to high-level semantics and enabling more stable, accurate gradient inversion in DQA models. We will include in-depth discussion in the appendix illustrating their importance.

Q5. "auxiliary priors scheduler.."

A5. Unlike prior works (e.g., GradViT), we found that using only ($L_{grad}$) in early iterations was insufficient for reconstructing meaningful text—likely due to the template reducing gradient signals in sensitive regions. To address this, we use a scheduler that starts with strong prior weights and gradually reduces them, stabilizing early optimization. The table below compares this to GradViT’s approach, which delays prior introduction until later iterations:

As shown, GradViT’s scheduler struggles to reconstruct meaningful content, especially in terms of semantic fidelity. We will include this comparison in the appendix and integrate the key findings into the final version of the paper.

Q6. "set hyperparameters.."

A6. The prior weights were manually set via grid search to optimize reconstruction performance. As noted in Answer 2, their absolute values don’t reflect relative importance due to differences in loss scales. Both Gaussian and TV priors are especially useful early in optimization, promoting smoothness and structure that complement sharper and semantic losses like Laplacian and text similarity.

Q7. "Notations inconsistency.."

A7. Thank you, this will be fixed in the final version.

We thank the reviewer for his time, effort, and valuable suggestions.

Q1. "access to a template.."

A1. While the use of a template may appear simplified at first glance, we argue that this setup is practically grounded and reflects real-world scenarios. In many document-based applications, such as invoices, medical forms, boarding passes, or receipts, the layout and structure are highly standardized across users, with only the sensitive fields (e.g., names, amounts, dates) varying. For example, a receipt issued by a vendor often follows the exact same format for all customers. If an attacker has previously received such a receipt or obtained a sample from another user, it effectively serves as a valid template when attempting to reconstruct another client’s document. This assumption is especially realistic in settings where the attacker is an insider (e.g., the server operator) or has access to public document templates shared across users. We will clarify this practical motivation in the final version of the paper.

Q2. "QA reconstruction.."

A2. We thank the reviewer for this important observation. The reconstruction of the question-answer (QA) pair is indeed a critical step in our pipeline, and we address this in Section 3.3. In our current work, we assume access to the QA pair, based on strong evidence from prior analytic-based methods—particularly DAGER (Petrov et al., 2024)—which showed that QA tokens can be accurately recovered from gradients in small-batch federated learning, achieving 100% recovery. Our focus is on the document reconstruction step conditioned on known QA pairs, reflecting a realistic threat model where such tokens are obtainable using existing techniques. To strengthen this connection, we will include the QA reconstruction in the final version of the paper and clarify the assumption more explicitly.

Q3. "How can PIIs.."

A3. Please see Reviewer fLco, Answer #5—specifically the visualization of the cross-attention maps, which illustrates how private data outside the question and answer can still influence the computed gradients, offering insight into how such regions are captured.

Q4. "Until page 5.."

A4. The sentence in question was intended to convey an intuition about the general difficulty of reconstructing small-sized text regions (such as names or dates), which typically occupy only 1–2% of the image area and thus contribute weakly to the overall gradient signal—especially when compared to larger visual elements like headers, tables, or logos. Importantly, our attack is able to reconstruct personally identifiable information (PII) in the document regardless of the specific question or answer, as long as the attack is performed on a model initialized with random weights.

Q5. "..threat model.."

A5. "Our threat model is consistent with most prior gradient inversion works, where the adversary is assumed to have access to the model architecture, parameters, optimizer, and input-output format—but not the sensitive input data itself. The only additional assumption in our setup is access to a template document, which reflects a realistic scenario in many applications (e.g., invoices, receipts, forms) where the document structure is shared across users and only a few fields contain private information. This assumption is grounded in practical document workflows."

Q6. "PIIs are already known.."

A6. While the answer in the QA pair may reveal a piece of PII, it typically represents only a small portion of the sensitive content (e.g., a single name, date, or value). Many other PII fields remain hidden and can still be reconstructed from gradients. Our attack can also reconstruct text outside the question and answer, due to the model’s cross-attention attending to broader regions—especially in early training, when attention is more diffuse. Additionally, see Reviewer fLco, Answer 6, for an experiment showing that PII can still be reconstructed even when the QA tokens are replaced with random tokens. We provide sample data, including documents and their corresponding Q-A pairs, here.

Q7. "results without templates.."

A7. Thank you for the question. We conducted an experiment where the attack was performed without access to a template, and found that reconstruction quality dropped significantly. Without the template, the model must infer both the layout and the content, making optimization far more difficult—especially in structured documents like receipts or forms. We see this work as a step toward full document reconstruction, and future work could explore removing the template assumption entirely.

We thank the reviewer for his time, effort, and valuable suggestions.

Q1. "found [r1].."

A1. We thank the reviewer for pointing out this work. [r1] explores gradient inversion in multi-modal FL using synthetic image-text pairs processed by separate, modality-specific models trained independently, with no shared representations. The attack is coordinated via mutual knowledge distillation. While valuable, we believe the use of the term “multi-modal” in [r1] is somewhat misleading, as it refers to parallel models that only interact during inversion—unlike standard multi-modal architectures that jointly fuse modalities (e.g., via cross-attention). Our work focuses on such deeply fused, jointly trained models like those used in DQA. Nonetheless, we appreciate this connection and will include a discussion of [r1] in the related work section.

Q2. "theoretical analysis.."

A2. We would like to clarify that our work follows a well-established line of research on gradient inversion, which has primarily focused on empirical demonstration of vulnerabilities rather than formal theoretical analysis. Foundational works such as DLG (Kim et al., 2019), iDLG (Zhao et al., 2020), and GradViT (Hatamizadeh et al., 2022) have aimed to develop practical attack strategies and assess reconstruction quality under various conditions. Given the complexity and high dimensionality of modern neural networks—especially in multi-modal setups like DQA—formal guarantees remain an open challenge. As in prior work, we rely on rigorous experimental evaluation across architectures and settings. Our work extends gradient inversion to multi-modal architectures, with ablations isolating each design component’s effect.

Q3. "FL setup.."

A3. We appreciate the comment and clarify that Federated Learning is a training paradigm, not a model type—it can be applied to any architecture, including Donut and LayoutLMv3. Our setup simulates gradients computed locally by a client on private data, which are then forwarded to a central server—where the adversary is assumed to reside. This is consistent with prior gradient inversion works and reflects a server-side threat model. Since our focus is on leakage from a single client, the total number of clients does not affect the core attack. We will revise the paper to clarify this setup.

Q4. "Priors Effect.."

A4. Please see Reviewer xopj, Answer 1.

Q5. "iteration t.."

A5. We thank the reviewer for this interesting and important question. In our experiments, the attack was performed at iteration t=0, using the model’s initial random weights. We also ran the attack at later training stages and found it to be significantly less effective. We hypothesize that in well-trained models, attention and token representations become sharply focused on regions relevant to the question and answer, which narrows the gradient signal. In contrast, early in training, encode more diverse and spatially distributed information, making reconstruction easier. To support this, we include a visualization of cross-attention maps for both randomly initialized and fine-tuned models, showing how attention becomes increasingly narrow and localized when the model is finetuned. Here, we present a systematic analysis of attack success across training iterations, confirming that inversion becomes progressively more difficult as the model converges.

Q6. "QA Effect.."

A6. Thank you for raising this point. As shown in Section 4.2.2, gradient inversion fails without access to both the question and answer tokens, highlighting the need for semantic grounding. However, when we say the specific content is irrelevant, we mean the attack depends on the token identities and positions, not their semantic meaning. To demonstrate this, we replaced the original QA with random tokens of the same length and structure. The reconstruction quality remained nearly identical:

This suggests that having access to the correct tokens—regardless of their meaning—is sufficient for the attack. An reconstruction example is shown here. We will clarify this distinction in the paper and provide additional examples.

Q7. "DLG and iDLG.."

A7. Yes, the results for DLG and iDLG in Table 1 are identical. The only difference is that iDLG has access to the answer (i.e., the ground-truth label). However, as shown in Section 4.2.2, the answer alone is insufficient for reconstruction in DQA, since it lacks the context provided by the question, which guides model attention and gradient flow. As a result, iDLG offers no advantage over DLG in our setting.

We thank the reviewer for his time, effort, and valuable suggestions.

Q1. "..confidence intervals.."

A1. Our original submission reported mean values, as we observed low variance across runs, confirming that our method is stable and consistently outperforms baseline methods. We will include the standard deviation values in the final version of the paper and provide additional details in the appendix to enhance transparency and reproducibility.

Q2. "..defences..utility.."

A2. Our proposed defense strategy applies perturbations to the input templates outside the FL training procedure. These perturbations are added to publicly available or previously observed templates and are used solely by the attacker as priors during the inversion process—not by the participating clients during training (which use "clean" documents). As a result, the actual training inputs, local computations, and shared gradients remain completely untouched, and the federated learning process proceeds as usual. This ensures that the utility of the multi-modal models is fully preserved, unlike defenses that modify gradients directly (e.g., via differential privacy), which often degrade model performance [1]. We will update the paper to clarify this distinction and emphasize the utility-preserving nature of our defense.

Q3. "..performance of GI-DQA.."

A3. We thank the reviewer for this thoughtful observation. To clarify, overfitting is not a factor in our experiments, as all models—both OCR-based (e.g., LayoutLMv3) and OCR-free (e.g., Donut)—are evaluated using randomly initialized weights (see Reviewer fLco, Answer 5). This ensures that performance differences arise from architectural properties, not training dynamics. We hypothesize that the variation in GI-DQA performance stems from differences in the size and structure of the visual backbones. LayoutLMv3 uses a smaller image encoder, leading to lower representational complexity and making gradient inversion more tractable. Donut, in contrast, uses a larger and more expressive visual backbone, which produces more abstract and distributed features—making inversion harder without strong priors.

We will revise the manuscript to clarify this architectural difference and its impact on attack performance.

Q4. "Differential Privacy (DP).."

A4. We fully agree that Differential Privacy (DP) is a well-established and rigorous approach for protecting client data in federated learning. While defense design is not the main focus of our work, we propose a lightweight input-level method that operates entirely outside the FL training loop, leaving shared gradients untouched. This makes it a practical alternative when DP integration may be too costly or degrade utility. To provide a point of comparison, we evaluated a DP-style defense by adding Gaussian noise to the shared gradients. Results are shown below:

While DP yields lower PSNR (better in terms of defense), our method more effectively disrupts text recovery, as shown by binary accuracy. This demonstrates the complementary nature of our defense, which focuses on degrading machine-readability, even if some visual structure remains. We will include this discussion and table in the final version of the paper.

Q5. "..use GradViT.."

A5. We originally intended to include GradViT as a baseline in Table 1, given its relevance to gradient inversion in vision models. However, the authors of GradViT did not release their code, and despite reaching out to them multiple times, we unfortunately did not receive a response. Given the complexity of accurately reproducing their method—particularly in the context of multi-modal models—we decided against including potentially inaccurate or incomplete re-implementations. We would be happy to include GradViT as a baseline in future versions if their code becomes available.

Q6. "..Gaussian.. in addition..Laplacian.."

A6. While Table 2 may suggest the Gaussian prior is less impactful than Laplacian, we include both as they address complementary aspects of reconstruction. The Gaussian prior promotes smoothness and stabilizes early optimization by reducing noise, while the Laplacian prior sharpens edges and preserves fine details—crucial for small text recovery. See Reviewer xopj, Answer 1 for the ablation of individual priors and Answer 4 for insights into the selection of priors.

[1] Cummings, R., Desfontaines, D., Evans, D., Geambasu, R., Huang, Y., Jagielski, M., ... & Zhang, W. Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment. Harvard Data Science Review, 6 (1), jan 16 2024.