MQFL-FHE: Multimodal Quantum Federated Learning Framework with Fully Homomorphic Encryption

Comment 1: Could you clarify whether the decryption and global model update process occurs on the server side or client side? The algorithm and implementation appear to show different approaches.

Response: The decryption and global model update occur on the client side. After the server performs homomorphic aggregation of the encrypted model updates, it sends the aggregated encrypted model back to all clients. Each client then decrypts the aggregated model using their private key and updates their local model accordingly. This ensures that the server never has access to the decrypted model parameters, preserving the privacy of the clients' data. We revised Algorithm 1 and the corresponding explanation in the manuscript to make this process clear and consistent.

Comment 2: Does your implementation require clients to share the same secret key? If so, how is this key sharing accomplished securely without server access to this information?

Response: Our implementation eliminates the need for clients to share the same private key by utilizing a public-key homomorphic encryption scheme. In this setup, all clients and the server share a common public key for encryption, while each client retains their own private key for decryption, which is never shared. This ensures that the server with access only to the public key, can perform homomorphic operations on encrypted data from multiple clients without being able to decrypt any of it. The key sharing process is securely managed through a trusted setup phase, where public and private keys are generated. The public key is distributed to all participants, while private keys remain securely with their respective clients. Clients encrypt their model updates using the shared public key and transmit the ciphertexts to the server via secure communication channels.

Comment 3: If your system employs secret-sharing schemes for key distribution among clients, how do you handle client dropout scenarios? Have you investigated the impact of client unavailability on the secret-sharing mechanism?

Response: Since clients do not share secret keys in our framework, client dropout does not affect the key distribution or the encryption and decryption processes. Each client operates independently with their own private key. If we were to use a secret-sharing scheme for key distribution, client dropout could pose challenges in reconstructing the secret key or maintaining the integrity of the system. However, this is not applicable in our implementation.

Comment 4: The paper claims quantum computing helps mitigate FHE noise accumulation. Could you provide a detailed mathematical analysis demonstrating how quantum computing specifically counteracts this noise?

Response: Thank you for raising this important point. In Appendix Section 7.1, we provide a detailed mathematical analysis demonstrating how quantum computing helps mitigate noise accumulation in FHE-based systems. The analysis highlights the inherent properties of quantum operations governed by the SU(2) group, which enforces unitary constraints that preserve the norm of quantum state vectors. This ensures that errors introduced during computations remain bounded, preventing the unbounded accumulation of noise typically observed in purely classical systems. Moreover, the oscillatory nature of quantum errors, arising from predictable angular deviations during state evolution, introduces a periodic "resetting" mechanism that naturally stabilizes noise growth over time. By incorporating quantum layers within hybrid FHE-quantum architectures, the bounded and oscillatory behavior of quantum noise helps counteract the accumulation of FHE-induced errors. The quantum layers periodically stabilize error propagation, and the subsequent classical processing further suppresses residual quantum noise. This hybrid approach leverages the strengths of both paradigms, with quantum layers ensuring controlled noise dynamics and classical layers refining the output for enhanced fidelity. These mechanisms collectively improve the robustness of encrypted computations, even in deep architectures.

Comment: The authors demonstrate the framework's performance using DNA and MRI datasets, but the scalability to other types of multimodal data is unclear. Could the authors elaborate on how adaptable the proposed approach is to other types of data and any challenges they foresee?  The paper discusses the impact of homomorphic encryption on model accuracy. It would be beneficial to expand on this discussion with additional citations to similar works and provide more thorough experiments to investigate the impact. For example, testing different hyperparameters or model structures.

Response: We thank the reviewer for their constructive feedback. Below, we provide clarifications and further details on the adaptability of our framework to other types of multimodal data and the impact of FHE on model accuracy. Regarding the scalability of our proposed approach to other types of multimodal data, we agree that this is an important point to address. In Section 7.3 (Efficacy of MQMoE Approach) of the appendix, we demonstrate the versatility of our framework by showing its compatibility with a diverse set of datasets and modalities, including text, image, audio, and video. Specifically, Table 6 illustrates the framework’s ability to handle various data types, such as CIFAR-10 (image), DNA sequences, MRI scans, and the RAVDESS dataset (audio-visual). The RAVDESS dataset, which includes synchronized emotional speech and song recordings, is an excellent example of how the MQMoE approach integrates both audio and visual modalities for emotion recognition. This provides strong evidence of the framework’s capability to manage complex multimodal tasks. In response to the comment on the impact of homomorphic encryption on model accuracy, we have extended our discussion in Section 7.2 (Effects of FHE Modifications). As outlined in Table 5, we provide a detailed analysis of how different encryption parameters (bit scale, polynomial modulus degree, and extrema count) affect both the number of encryptable values and computational efficiency. We highlight that reducing parameters such as the bit scale and polynomial modulus degree increases the number of encryptable values but at the expense of lower security levels. This trade-off is crucial when considering the impact of FHE on model accuracy and efficiency. We acknowledge that further experiments are necessary to better understand these trade-offs, and we plan to extend our work by testing different hyperparameters and model structures to investigate how these modifications influence model performance.

Comment 1: In the experimental section, the paper lacks information on the parameter configuration of the CKKS scheme and its corresponding security levels, which may affect the reproducibility and practicality of the experimental results.

Response: We agree that specifying the parameter configurations of the CKKS scheme is crucial for reproducibility and assessing security levels. In our experiments, we used the following parameters:

• Polynomial Modulus Degree (n): 8192. This parameter determines the ring dimension and directly affects both security and computational efficiency. A degree of 8192 offers a balance between performance and security, providing approximately 128 bits of security based on standard lattice-based cryptography estimates.

• Coefficient Modulus (q): A modulus chain composed of four primes with bit-lengths [60, 40, 40, 60], totaling 200 bits. This configuration supports the required multiplicative depth for our computations while maintaining manageable ciphertext sizes.

• Scaling Factor ($\delta$): $2^{40}$. The scaling factor is chosen to balance precision and noise growth during homomorphic operations, ensuring that approximate arithmetic remains accurate throughout the computations.

These parameters balance security and computational efficiency, offering approximately 128 bits of security based on standard estimates for lattice-based cryptography. Including these details ensures that others can reproduce our experiments and understand the security implications of the parameter choices. We have provided these specifics in the ``Methodology'' section of the manuscript.

Comment 2: The layout of some figures in the paper could also be improved, as some images are too small to read comfortably.

Response: We appreciate your feedback on the figure layouts. We have ensured that all figures are appropriately sized and clearly presented to enhance readability. High-resolution images have been used to ensure that all details are readable.

Comment 3: How do homomorphic encryption, quantum computing, and federated learning work together, and how are their respective advantages reflected within the framework?

Response: In our framework, HE, QC, and FL are integrated to create a secure and efficient system for training machine learning models on decentralized and sensitive data. FL allows multiple clients to collaboratively train a global model without sharing their raw data, thus preserving data privacy and complying with regulations like GDPR. However, the exchanged model updates in FL can still be vulnerable to inference attacks. To enhance privacy, we incorporate FHE, specifically the CKKS scheme, which enables computations on encrypted data without decryption. By encrypting the model updates prior to transmission, FHE ensures that the server can aggregate the encrypted updates to form a global model without accessing the underlying plaintext data, which addresses the privacy concerns in FL by protecting model updates from potential eavesdropping or malicious actors. However, FHE introduces significant computational overhead due to the complexity of performing operations on encrypted data, which can degrade model performance and increase training times. To mitigate this issue, we integrate QC within our model architecture, which potentially offers computational advantages for certain types of operations and can enhance the expressive power of neural networks through QNNs. By integrating these quantum layers into our model, we aim to counteract the performance degradation caused by FHE. In our MQMoE model, QC is employed to process different data modalities, such as MRI images and DNA sequences. Each modality is handled by specialized quantum circuits acting as experts, capturing complex patterns and correlations that might be challenging for classical models. The outputs of these quantum experts are then combined using a gating mechanism, which dynamically weights their contributions based on the input data. Our framework's combination of FHE, QC, and FL allows us to maintain strong privacy guarantees without sacrificing model performance. Homomorphic encryption secures the data during transmission and aggregation, FL enables collaborative model training without data centralization, and QC enhances computational efficiency and model expressiveness. The respective benefits of each technology are reflected in the framework through improved data privacy, efficient handling of encrypted computations, and the ability to learn from complex, multimodal datasets in a distributed and secure manner.

Comment 1: The experimental setup is quite vague. It is unclear what is the distribution of and size of the training datasets for each client. One of the key issues in FL is heterogenous setups, that is, different clients may have different distribution and different sizes of training dataset. Authors should evaluate the approach in heterogenous settings.

Response: Thank you for highlighting the need for more clarity in our experimental setup. In our experiments, we distributed the datasets equally among clients to focus on the fundamental effects of integrating QC and FHE in FL. Each client received an equal portion of the training data, ensuring uniform data size and distribution. We agree that evaluating the approach in heterogeneous settings is important, as real-world FL scenarios often involve clients with varying data distributions and sizes. In future work, we plan to extend our experiments to include heterogeneous data distributions, such as non-IID data, and clients with different amounts of data to assess the robustness and adaptability of our approach under more realistic conditions.

Comment 2: Is the MQMoE framework designed for paired data, Fig 3? That is does the model take (MRI,DNA) pair as the input in the multimodal experiments? If the dataset is paired, how do you handle missing data? Table 1 has different number of MRI & DNA data, so there may be some missing datasets?

Response: Yes, our MQMoE framework is designed to handle paired data. In our multimodal experiments, the model takes pairs of MRI images and corresponding DNA sequences as input. Each modality is processed through its specialized expert network within the MQMoE architecture, allowing the model to learn complementary representations from both modalities for the same subject. To address cases where data for one modality may be missing (e.g., an MRI without a corresponding DNA sequence), the framework employs masking strategies during training and inference. Specifically, when a modality is unavailable, its expert network is skipped, and the MQMoE dynamically re-weights contributions from the available modalities, ensuring robust performance. As noted in Table 1, while there are mismatched numbers of MRI and DNA samples due to incomplete datasets, our framework is equipped to handle such scenarios without compromising multimodal integration.

Comment 3: The test accuracy in multimodal setup is poorer than unimodal case in Table 1? So does this suggest that there is no benefit of having an additional modality?

Response: Thank you for your observation. While it may seem that the test accuracy in the multimodal setup is slightly lower than in the unimodal case, the difference is minimal, as highlighted in Table 4. For example, in the QFL + FHE configuration, the test accuracy for the multimodal setup (DNA: 95.31%, MRI: 87.26%) is comparable to the unimodal DNA (94.32%) and MRI (88.25%) accuracies. These minor variations are expected due to the increased complexity of integrating multiple modalities. However, the multimodal setup provides significant advantages beyond test accuracy.

Specifically, a multimodal pipeline allows for a unified inference process, which is computationally more efficient compared to managing two separate unimodal architectures. Running independent pipelines for DNA and MRI would require duplicating resources and increasing inference time, whereas the multimodal setup consolidates both modalities into a single framework, streamlining the inference process. Additionally, the multimodal approach enhances robustness by leveraging complementary information from both modalities, which can be particularly beneficial in real-world scenarios where a single modality might be insufficient. Thus, the multimodal framework provides both practical and performance-related benefits.

Comment 1: Quantum computers are inherently more noisy due to its probabilistic nature. Why would utilizing QC “reduce” noise accumulation in conjunction with FHE? I could not find any intuition or explanation.

Response: Thank you for the insightful question. We have addressed this concern by including a detailed explanation in Appendix Section 7.1: Mathematical Analysis of Error Propagation and Noise Stabilization in FHE-Quantum Hybrid Models. In this section, we explain how QC, despite its inherent noise due to the probabilistic nature of qubits, can actually help reduce noise accumulation when combined with FHE. QC processes information using qubits confined to the Bloch sphere, with operations applied through norm-preserving unitary transformations. This limits error propagation in QML when compared to classical ML models, where errors tend to accumulate across computations. QC, when combined with FHE, helps contain the noise introduced by FHE. This is achieved through the periodic reset of error propagation mechanisms in QML. This combination ensures that noise does not significantly affect the accuracy of the computations, leveraging the quantum properties for error containment and FHE for secure, noise-resistant computation.

Comment 2: It seems that each client is also assumed to have access to a quantum computer, based on Eq. in line 199. Encoding classical data to a quantum state, especially if the data is multi-modal, sounds quite non-trivial. Can you comment on how this can be achieved?

Response: Yes, encoding classical, especially multimodal, data into quantum states is non-trivial. In our work, we assume that clients have access to quantum resources, which is increasingly feasible through cloud-based quantum computing platforms (e.g., IBM Quantum Experience, Amazon Braket). For encoding multimodal data, we expanded Section 4.2 of the manuscript to provide a detailed methodology. Specifically, our approach leverages the PrepareMultimodalDataset function to preprocess and transform data from distinct modalities into a form suitable for hybrid classical-quantum processing. For example:

• DNA Sequences: These are split into k-mers to capture sequence patterns and then converted into a textual representation. This textual data is subsequently vectorized using TF-IDF (Term Frequency-Inverse Document Frequency), which provides a sparse numerical representation.
• MRI Images: These are preprocessed using standard transformations such as resizing, normalization, and noise reduction to ensure compatibility across training pipelines.

Once prepared, these datasets are paired to represent multimodal inputs. The next step involves processing the encoded data using a hybrid classical-quantum model. In this model, the classical layers preprocess the data to feed it into the quantum layers, which provide advantages such as higher-dimensional feature representation and enhanced expressivity. This process is facilitated using parameterized quantum circuits (PQC).

We recognize that some of these processes are complex. However, each step follows well-documented and validated methodologies in existing literature [1-7]. Our contributions lie in integrating these components into a unified framework to tackle the specific challenge of performance degradation due to FHE in FL.

• [1] Schuld, Maria, Ryan Sweke, and Johannes Jakob Meyer. "Effect of data encoding on the expressive power of variational quantum-machine-learning models." Physical Review A 103.3 (2021): 032430.

• [2] Huang, Hsin-Yuan, et al. "Power of data in quantum machine learning." Nature communications 12.1 (2021): 2631.

• [3] Ovalle-Magallanes, Emmanuel, et al. "Quantum angle encoding with learnable rotation applied to quantum–classical convolutional neural networks." Applied Soft Computing 141 (2023): 110307.

• [4] LaRose, Ryan, and Brian Coyle. "Robust data encodings for quantum classifiers." Physical Review A 102.3 (2020): 032420.

• [5] Innan, Nouhaila, and Muhammad Al-Zafar Khan. "Classical-to-Quantum Sequence Encoding in Genomics." arXiv preprint arXiv:2304.10786 (2023).

• [6] Wang, Aijuan, et al. "HQNet: A hybrid quantum network for multi-class MRI brain classification via quantum computing." Expert Systems with Applications 261 (2025): 125537.

• [7] Senokosov, Arsenii, et al. "Quantum machine learning for image classification." Machine Learning: Science and Technology 5.1 (2024): 015040.