Enhancing Privacy in Multimodal Federated Learning with Information Theory

Weakness1: Lack of details on the implementation of the conditional mutual information calculation

Response1.1: We sincerely thank the reviewer for this insightful question regarding our CMI implementation. The conditional mutual information $I(G;D^{i}|D^{-i})$ in Equation (8) is actually computed using the Neural Estimators for Conditional Mutual Information framework (Molavipour et al., 2021). To address heterogeneous gradient dimensions across modalities, our implementation: 1. Flattens each modality's gradient tensor into a 1D vector 2. Concatenates all modality gradients into a unified representation $\tilde{G} = \text{concat}(\text{vec}(G^{\text{text}}), \text{vec}(G^{\text{img}}), \cdots)$ 3. Processes three distinct inputs through the MLP classifier $f_\theta$: - $\tilde{G}$ (concatenated gradients) - $D^i$ (target modality data) - $D^{-i}$ (complementary modalities data) . The classifier distinguishes: 1.True pairs$(\tilde{G}, D^i)$ with correct $D^{-i}$ 2.Counterfactual pairs: $(\tilde{G}, \tilde{D}^i)$ where $\tilde{D}^i$ is shuffled within k-nearest neighbors of $D^{-i}$ . The final CMI value is derived as: $\hat{I} = \frac{1}{B}\sum_{b=1}^B \log \frac{f_\theta(G_b, D_b^i|D_b^{-i})}{f_\theta(G_b, \tilde{D}_b^i|D_b^{-i})}$ .

Weakness2: Applicability to audio-video modalities

Response1.2: We deeply appreciate this important question about modality generalization. Sec-MMFL extends to audio-video systems through its architecture-agnostic framework. For audio inputs, we process log-Mel spectrograms using Resnet-18 encoders, while video inputs utilize 3D-ResNet encoders for temporal feature extraction. The conditional mutual information estimation follows identical neural estimation procedures as in image-text systems, with protection strengths automatically calibrated based on modality-specific leakage risks. Our validation on CREMA-D demonstrates effecness: leakage risk assessment automatically detects audio's higher vulnerability (38% greater $R_i$ than video).

Weakness3: Lack of SOTA comparisons for MMFL

Response1.3: Thanks for the question. To strengthen validation, we have supplemented experiments with FedMultimodal [9], a state-of-the-art MMFL baseline, on CrisisMMD dataset, and Hateful Memes dataset. The table below presents a comprehensive comparison of all methods in terms of task performance and defense capability across two datasets. While FedMultimodal (specifically designed for MMFL) achieved marginally higher task accuracy on both datasets, it lacks dedicated defense mechanisms. In contrast, our Sec-MMFL method maintains comparable task performance while demonstrating significantly more balanced privacy protection across all modalities. We will add these results to Table 3 and explicitly discuss FedMultimodal comparisons in Section 5.3 of the revised manuscript.

Suggestions: Language revisions

Response1.4: Thank you so much for the suggestions. All suggested language improvements have been implemented: client data access description (line 125) updated to "their local private dataset", subject-verb agreement in GIA description (line 138) corrected, and complex sentence (line 145) simplified to "Figure 2 illustrates the flow of data from each MMFL modality to attackers via GIA" for enhanced readability.

Question1:Could you provide more details on the specific implementation of the conditional mutual information calculation?

Response2.1: Thanks for the question. We have already explained in Response1.1 the details on the specific implementation of the conditional mutual information calculation using the neural estimators for conditional mutual information method.

Question2: How might Sec-MMFL be adapted for other modality combinations, like audio-video modalities? Is it possible to outline a potential approach using existing datasets like CREMA-D?

Response2.2: Thanks for the question. Our framework naturally extends to CREMA-D through modality-specific feature extractors: Resnet-18 for log-Mel spectrograms (audio) and 3D-ResNet for optical flow (video). Details are explained in Response1.2.

Question3: Would it be possible to include brief benchmarking against recent MMFL defenses like FedMultimodal?

Response2.3: Thank you for this inquiry. We have supplemented experiments in Response1.3.

Question4: For GIA attacks, how are pseudo inputs initialized? When applying LDP noise, did you clip gradients before or after adding noise?

Response2.4: We appreciate the question. For gradient inversion attacks, pseudo-inputs are initialized with $\mathcal{N}(0,1)$ for images and random vocabulary tokens for text. Regarding LDP implementation, gradients are clipped (norm=1.0) before adding Gaussian noise . Optimization uses Adam (3000 iterations, lr=0.1) with total variation regularization ($\lambda=0.$2) to balance attack realism with practical constraints.

Weakness1: The privacy protection effectiveness is mainly measured by PSNR and TRR, which might not fully capture the privacy implications in complex real-world scenarios. More comprehensive metrics could strengthen the evaluation.

Response1.1: We appreciate this critical observation regarding evaluation metrics. To provide more comprehensive privacy assessment, we will supplement our evaluation additional metrics like SSIM (Structural Similarity Index) and CLIP-space.

Weakness2: There is limited exploration of the impact of different model architectures on the performance of Sec-MMFL. The framework's effectiveness might vary depending on the choice of encoders and fusion mechanisms.

Response1.2: Thank you for highlighting this important dimension. We will expand Section 5 with comprehensive architecture sensitivity analysis including: 1) Transformer-based encoders (BERT/ViT) versus CNN/RNN architectures; 2) Different fusion mechanisms (concatenation vs attention-based); and 3) Model complexity scaling from 1M to 100M parameters.

Weakness3/Question3: Omits discussion about potential risks - adaptive noise profiles could themselves leak modality sensitivity, creating new attack surfaces.

Response1.3: We appreciate this important security consideration. The fundamental protection mechanism ensures that reverse-engineering noise profiles is both practically infeasible and mathematically ineffective. Attackers face critical barriers: 1. Attackers only observe the final perturbed gradients $G' = G + \eta$ (where $\eta \sim \text{Gaussian}(0, \Delta f/\epsilon_i)$), lacking access to the original gradients $G$ or the specific noise parameters $\epsilon_*i$. This obscures the relationship between noise magnitude and modality sensitivity. 2. Even if attackers could hypothetically estimate relative noise levels, identifying which modality received stronger protection (indicating higher risk) provides no exploitable pathway. The local differential privacy transformation satisfies $(\epsilon,\delta)$-LDP guarantees (Eq.7), making the noise-added gradients statistically irreversible.

Weakness4: Terminology Revisions.

Response1.4: We thank the reviewer for catching these inconsistencies - all instances will be corrected: Line 268: "information leakage risk among different modalities"; Line 286: "text modality protection"; Line 288: "two modalities more balanced". Comprehensive terminology standardization will be implemented throughout Sections 3-5 to ensure consistent use of "modality" when referring to data types.

Question1: The paper mentions the use of early and late fusion strategies in the experiments. Could you explain how the proposed method's effectiveness might vary depending on the fusion strategy used?

Response1.1: We appreciate this fundamental question. Sec-MMFL's effectiveness varies significantly with fusion strategies due to inherent information flow differences: Late fusion (feature-level integration) exhibits higher initial leakage risks (Table 1: R_text=0.5383 vs 0.2539) but enables precise modality-specific protection through our noise allocation, yielding maximum 12% privacy gains. In contrast, early fusion creates entangled representations where modality-specific protection is less effective but baseline risks are lower. Our method achieves optimal balance in late fusion scenarios where modality-specific vulnerabilities are most pronounced and conditional mutual information can be precisely quantified.

Question2: The paper mainly uses LeNet-5 and TextCNN as encoders. How would the choice of more complex or different model architectures affect the performance of Sec-MMFL in terms of both privacy protection and model utility?

Response1.2: We appreciate the inquiry. Complex architectures demonstrate two countervailing effects: 1) Increased baseline vulnerability due to richer gradient information (higher $R_i$ values); but 2) Enhanced noise absorption capacity enabling lower noise injection for equivalent protection. Sec-MMFL's architecture-agnostic design ensures relative improvements over uniform protection actually increase with model complexity.

Thank you for the response. It has addressed my concerns, and I will maintain my score.

Weakness1: The paper lacks sufficient mathematical rigor in key derivations. For instance, there is no clear explanation of how the allocation scheme in Equation (10) guarantees that the total privacy budget equals $\epsilon$.

Response1.1: We sincerely appreciate you for pointing out the mathematical formulation problem in Equation (10). This resulted from a typographical error in the manuscript where the normalized risk weight $R_i'$ was incorrectly placed in the denominator. The allocation scheme and all experiments actually implemented the formulation:$\epsilon_i = \epsilon \times R_i'$. This scheme is mathematically sound because the normalization property of $R_i'$ (defined in Equation (9) as: $R_i' = \frac{e^{-R_i}}{\sum_{t=1}^M e^{-R_t}}$inherently satisfies: $\sum R_i' = 1$ which guarantees: $\sum \epsilon_i = \epsilon$ through the relation: $\sum_{i=1}^M (\epsilon \times R_i') = \epsilon \times \underbrace{\sum_{i=1}^M R_i'}_{=1} = \epsilon$. We will correct Equation (10) in the manuscript and explicitly prove budget conservation in Section 4.2.

Weakness2: How the gradient mutual information in Equation (12) is computed across heterogeneous network structures.

Response1.2: Thanks for raising the concern. We address cross-modal gradient mutual information computation across heterogeneous encoders using Mutual Information Neural Estimation (MINE) method with Donsker-Varadhan representation. The gradients from different modalities are flattened into 1-dimensional vectors, concatenated, and then fed into a three-layer MLP discriminator network (layer sizes [32, 64, 32] with ReLU activation). The model was trained for 2000 epochs using a learning rate of 1e-3. Raw text encoder gradients $G^{text} \in \mathbb{R}^{N \times d*_{text}}$ and image encoder gradients $G^{image} \in \mathbb{R}^{N \times d_*{image}}$ serve as direct inputs. Positive sample pairs ${(G_i^{text}, G_i^{image}) for i=1 to N}$ are constructed from gradients originating from identical data instances to represent the joint distribution $P_{G^{text},G^{image}}$, while negative sample pairs${(G_i^{text}, G_{\pi(i)}^{image}) for i=1 to N}$ are generated via random permutation $\pi$ of image gradients to simulate the marginal product distribution $P_{G^{text}} \otimes P_{G^{image}}$. The mutual information is computed as: $I(G^{\text{text}};G^{\text{image}}) \geq \sup_\phi \big( E_{P_(joint)}[T] - \log E_{P_(marginal)}[e^T] \big)$. So this approach handles architectural heterogeneity because the critic network $T_\phi$ learns relationships in the unified space independently of source architectures.

Weakness3: The paper would benefit from more comprehensive experiments incorporating state-of-the-art privacy attacks to strengthen the validity of the proposed defense.

Response1.3: Thank you very much for the advice. Regarding the attack methods, we followed the same experimental setup as Mgia in the MMFL security-related paper [20]. Indeed, more attack methods could be incorporated. In the following experimental setup, we compared the results of LDP-Fed and Sec-MMFL under the "See through gradients" attack [40], and we will supplement additional experimental results with more configurations and attack methods in future work.

CIFAR-10, $\epsilon$ =1:

CIFAR-100, $\epsilon$ =20:

CrisisMMD:

Hateful Memes:

Weakness4: From Table 2, it seems that the improvement over the existing approaches is rather marginal. The improvement in privacy is obtained with a degradation in test accuracy.

Response1.4: Thank you so much for raising this concern. We want to clarify that Sec-MMFL's primary contribution is achieving optimized privacy-utility equilibrium in multimodal federated learning, particularly addressing the inherent tension between protecting high-risk modalities (e.g., text) and preserving low-risk modality utility (e.g., images). While absolute accuracy gains may seem limited, our method demonstrates significant qualitative advances: Under equivalent privacy budgets (ε=1), Sec-MMFL reduces Text Recovery Rate (TRR) by 33% (0.61 vs. LDP-Fed's 0.91) while limiting accuracy degradation to just 1.2% (0.7801 vs. 0.7927) on CIFAR-10. This balanced protection is visually evidenced in Figure 1, where uniform defenses fail catastrophically—text is perfectly reconstructed while images remain irrecoverably blurred—whereas our conditional mutual information framework (Eq. 8) dynamically calibrates noise to prevent such asymmetric vulnerabilities. The critical innovation lies in transcending the "high TRR or low accuracy" dichotomy: At ε=2, Sec-MMFL maintains near-identical accuracy (90.01% vs. 90.05%) while slashing TRR by 26% (0.69 vs. 0.93), demonstrating unprecedented multimodal harmony. This balance is further validated in CrisisMMD results (Table 3), where our approach simultaneously improves LPIPS by 86% (1.085 vs. 0.583) and TRR by 34% (0.53 vs. 0.80) without compromising classification accuracy.

Weakness5: There are some typos. For example, in lines 253-254, the text encoder should be TextCNN, and the image encoder LeNet-5.

Response1.5: We sincerely apologize for this oversight. We will correct these typos in the paper and conduct another thorough check.

Question1: How is the entropy of gradients computed in Equation (8)? Given that encoders across different modalities have distinct architectures, the resulting gradients will have different dimensions. How are these heterogeneous gradients aggregated or represented when computing entropy across all modalities?

Response2.1: We sincerely appreciate the question. The conditional mutual information $I(G;D^{i}|D^{-i})$ in Equation (8) is actually computed using the Neural Estimators for Conditional Mutual Information framework (Molavipour et al., 2021). To address heterogeneous gradient dimensions across modalities, our implementation: 1. Flattens each modality's gradient tensor into a 1D vector 2. Concatenates all modality gradients into a unified representation $\tilde{G} = \text{concat}(\text{vec}(G^{\text{text}}), \text{vec}(G^{\text{img}}), \cdots)$ 3. Processes three distinct inputs through the MLP classifier $f_\theta$: - $\tilde{G}$ (concatenated gradients) - $D^i$ (target modality data) - $D^{-i}$ (complementary modalities data) . The classifier distinguishes: 1.True pairs$(\tilde{G}, D^i)$ with correct $D^{-i}$ 2.Counterfactual pairs: $(\tilde{G}, \tilde{D}^i)$ where $\tilde{D}^i$ is shuffled within k-nearest neighbors of $D^{-i}$ . The final CMI value is derived as: $\hat{I} = \frac{1}{B}\sum_{b=1}^B \log \frac{f_\theta(G_b, D_b^i|D_b^{-i})}{f_\theta(G_b, \tilde{D}_b^i|D_b^{-i})}$ . This architecture resolves dimensional heterogeneity through gradient flattening and concatenation, and conditional mutual information is estimated using neural networks rather than directly computing the entropy.

Question2: How does the allocation scheme in Equation (10) guarantee that the total privacy budget equals $\epsilon$ ?

Response2.2: Thanks for the question. We have already explained in Response1.1 the typographical error of Equation (10) and provided the corresponding detailed proof that guarantees the total privacy budget equals $\epsilon$ .

Question3: Similar to Question 1, given the difference in gradient dimensions across modalities, how is the mutual information between different modalities calculated in Equation (12)? What preprocessing or alignment steps are applied to enable meaningful mutual information computation?

Response2.3: Thank you for your question. We have already elaborated on similar computational procedures in Response 1.2. The gradients from different modalities only need to be flattened and fed into the neural network for computation via the Mutual Information Neural Estimation (MINE) method.

Question4: What is the rationale for selecting the hyperparameter λ? The paper appears to lack discussion on λ's sensitivity analysis or selection criteria in the experimental section.

Response2.4: Thank you for your question. Regarding the selection of hyperparameter λ, we acknowledge that the article lacks detailed discussion. In practice, we employed a grid search approach, testing multiple values (0.01, 0.05, 0.1, 0.3, 0.5, 1.0) and ultimately found that λ = 0.1 achieves an optimal balance between privacy preservation and task performance. Other values, while providing comparable privacy protection, incurred significantly greater accuracy degradation.

Question5: Why were the most recent attack methods, such as those referenced in [40], not included in the experimental evaluation?

Response2.5: Thank you for your question. We have addressed this question and supplemented relevant experiments in Response 1.3, and we plan to conduct more comprehensive experiments in future work.

I appreciate the authors for their efforts addressing my comments. I have several remaining questions.

1. For response 1.1, since the code is not available, there is no way to confirm the implementation is correct. That being said, I assume this is indeed a typo in my evaluation. Could the authors provide the rationale behind that $\sum_{i}\epsilon_{i} = \epsilon$ guarantees an overall privacy of $\epsilon$? Why is it linear?
2. In the response 1.2, I got the point that a MLP from prior work is used to estimate the mutual information between different modalities. But it is still not clear to me: (1) how is the model trained? Is it supervised? If yes, how to get the true label and the loss? (2) I do not fully get the equation $I(G^{text};G^{IMAGE})$ in the response, could you further elaborate? Please discuss the notations explicitly.
3. Does the network for mutual information estimation need to be trained for different datasets? Is this practical in FL considering the resource limitation and privacy concerns?

Overall, the paper needs a major revamp for the general readers to understand and appreciate the proposed method.

I would like to thank the authors for their efforts in addressing my questions.

Regarding the first question. The paper mainly considers approximate differential privacy according to Section 3.4 (your response above discusses pure DP, which differs from the manuscript and is confusing), and I assume the Gaussian mechanism is applied. I expect the authors to explicitly discuss how approximate DP is achieved, since many DP mechanisms in the literature guarantee approximate DP.

The authors seem to utilize the basic composition theorem of approximate differential privacy, which renders $\epsilon$ for the overall privacy linear with respect to each modality. Then how about $\delta$? How large are $\delta$'s in the experiments? More importantly, the basic composition theorem is not tight enough, which makes the results less convincing.

Question1: The paper highlights that "different modalities require varying protection strengths" as a key motivation, but what is the specific solution and validation of this problem? Can conditional mutual information guarantee such a balance across modalities? More qualitative analyses on this issue would likely facilitate an intuitive understanding of the proposed model.

Response1: Thank you for highlighting this crucial aspect. Our solution Sec-MMFL directly addresses modality-specific protection through two core innovations: 1) Using conditional mutual information $I(G;D^i \mid D^{-i})$ (Eq. 8) to quantify the unique leakage risk per modality, accounting for inter-modal correlations. This allows precise risk assessment where text modality consistently shows higher risk than images, validating its sensitivity to modality characteristics. 2) Adaptive noise allocation via Eq. 10 that inversely scales privacy budget $\epsilon_i$ with normalized risk $R'_i$ ensures high-risk modalities (e.g., text) receive stronger protection (lower $\epsilon_i$) while preserving utility for low-risk modalities. Experimental validation in Table 2 ($\epsilon=0.5$: TRR↓ 0.67 for text vs PSNR↓ 12.62 for images) demonstrates balanced protection. Qualitative analysis in Figure 5 visually confirms reduced cross-modal leakage, showing attackers cannot leverage recovered text to reconstruct high-fidelity images under Sec-MMFL, whereas uniform protection (LDP-Fed) allows clear image reconstruction when text is compromised. Additional parameter sensitivity analysis (Figure 4) further verifies stability across client counts and heterogeneity levels.

Question2/Weakness1: The current comparative methods do not adequately cover multimodal baselines. For instance, methods like LDP-Fed included in the comparisons are not designed for multimodal scenarios, making it difficult to effectively validate the advantages of the proposed method in MMFL settings.

Response 2: We appreciate this insightful suggestion. To strengthen validation, we have supplemented experiments with FedMultimodal [9], a state-of-the-art MMFL baseline, on CrisisMMD dataset, and Hateful Memes dataset. The table below presents a comprehensive comparison of all methods in terms of task performance and defense capability across two datasets. While FedMultimodal (specifically designed for MMFL) achieved marginally higher task accuracy on both datasets, it lacks dedicated defense mechanisms. In contrast, our Sec-MMFL method maintains comparable task performance while demonstrating significantly more balanced privacy protection across all modalities. We will add these results to Table 3 and explicitly discuss FedMultimodal comparisons in Section 5.3 of the revised manuscript. Furthermore, we will continue to incorporate additional mainstream multimodal methodologies to rigorously clarify the performance boundaries and comparative advantages of the proposed approach.

Question3: The model framework (Figure 3) is hard to follow. It is claimed that "the Noise Scale Adjustment module adjusts the privacy budget assigned to each modality, achieving a better balance between privacy protection and model effectiveness," yet it does not seem to illustrate how this balance is achieved.

Response 3: Thank you for this valuable feedback. We will revise Figure 3 to explicitly visualize the balancing mechanism: 1) Add directional arrows showing the flow from Leakage Risk Estimator (outputting $R_i)$ to Noise Scale Adjustment, where Eq. 9-10 convert $R_i$ into $\epsilon_i$ values (e.g., text: ($\epsilon_{\text{text}}$=0.2$\epsilon$), image: $(\epsilon_{\text{image}}=0.8\epsilon)$ when $(R_{\text{text}} > R_{\text{image}}$)). 2) Include gradient icons with variable noise magnitudes (larger noise symbols for text, smaller for images). This clarifies how high-risk modalities induce lower $\epsilon_i$ (stronger noise), while low-risk modalities retain higher $\epsilon_i$ (weaker noise) - precisely balancing protection and utility. The empirical evidence of this balance is demonstrated in Table 2 (e.g., at $\epsilon=1$, text TRR improves by 26% and image PSNR degrades only 6% vs LDP-Fed).

Response: We sincerely appreciate your valuable feedback regarding the sufficiency of multimodal method comparisons. We fully acknowledge the concern that supplementing only FedMultimodal as a comparative baseline might appear limited. This situation reflects a fundamental research gap in the MMFL field: while numerous studies focus on improving multimodal fusion accuracy (e.g., [9,25,45] in our references), very few have explored dedicated privacy preservation mechanisms tailored for multimodal scenarios. Several recent works have introduced novel defense mechanisms targeting entirely new threat scenarios without having directly comparable prior baselines. For example, SafeSplit (NDSS 2025) is the first defense designed for client‑side backdoor attacks in Split Learning. Existing defenses from other distributed frameworks like Federated Learning are not applicable, and there is a lack of effective backdoor defenses specifically designed for Split Learning. Therefore, they adopt conventional unified defense strategies like KRUM against backdoor attacks and apply them to the Split Learning setting as baselines for comparison. Similar baseline setting strategies are also adopted in works such as Dual Defense (NeurIPS 2024). In studies exploring differential privacy mechanisms in new scenarios, it is common to directly incorporate standard differential privacy as a comparative baseline, as in NetDPSyn (IMC2024). As the research on defense methods under MMFL is still in its early stage and lacks established baselines, we follow a similar comparative evaluation setup in our work.

To address this limitation comprehensively, we have conducted substantial additional experiments supplementing two state-of-the-art MMFL frameworks - FedSea: Federated Learning via Selective Feature Alignment for Non-IID Multimodal Data ( IEEE TRANSACTIONS ONMULTIMEDIA 2023) and HAMFL: Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning (CVPR 2024) . Both were adapted with equivalent basic differential privacy mechanisms (δ=1e-5, sensitivity=1.0) under identical attack scenarios to ensure fair comparison. The complete experimental results now demonstrate consistent trends across five privacy methods and three MMFL-specific baselines:

These extended results reveal three patterns: First, all MMFL+DP variants (FedMultimodal, FedSea, HAMFL) exhibit consistently high text leakage, confirming that even advanced multimodal architectures remain fundamentally vulnerable to gradient inversion attacks when using generic protection. Second, Sec-MMFL achieves higher LPIPS scores, indicating stronger protection for visual content - particularly crucial given images' higher sensitivity. Third, while specialized MMFL methods show marginally better accuracy, they do not have an advantage in terms of a more balanced level of privacy protection.

We sincerely appreciate your recognition, and your feedback has been invaluable in improving the clarity and quality of our work.