Exploring Modality Collaboration with Modality-Agnostic Transformers in Multi-Modal Federated Learning

We sincerely appreciate the effort and attention you've devoted to reviewing our paper. Your thoughtful feedback and the high rating have been incredibly helpful and encouraging.

The proposed framework is based on FedAVG, can other model aggregation methods be applied to further improve the performance?

As highlighted in Appendix A, while our FedCola framework is indeed grounded in the FedAVG algorithm, its essence extends beyond a mere aggregation method. FedCola encompasses a system-level approach, incorporating elements such as model architecture, parameter-sharing strategy, and temporal modality arrangement. Importantly, it is designed to be adaptable and compatible with a variety of model aggregation methods, not limited to FedAVG. This flexibility allows for potential enhancements in performance through the application of alternative aggregation techniques.

... what about the resources compared with CNNs?

In the experiments conducted for our study, we employed a ViT-Small model as the backbone, which comprises approximately 21 million parameters. This is comparable to the parameter size of a ResNet50, which contains around 24 million parameters. Therefore, the resource utilization between these two models is similar. However, it's important to note that Convolutional Neural Networks (CNNs) are not an ideal choice for our Modality Collaborated Federated Learning (MCFL) framework. This is because CNNs do not encode multi-modal information as effectively as transformers, which is a crucial capability for the objectives of MCFL.

During the warm-up stage, are participating clients sampled from only one modality?...

During the warm-up stage of our experiments, we ensured fairness by sampling clients uniformly across all modalities, employing a fixed random seed for all methods. It's important to clarify that while clients from all modalities are considered during the sampling process, communication during this stage is restricted to those with the specified warm-up modality. This approach means that the number of clients actively participating in communication is actually reduced in the warm-up stage.

As a result of this selective communication with fewer clients, the communication costs associated with the warm-up stage, as depicted in Figure 6, are lower compared to other stages. This reduced cost is a direct consequence of the limited client engagement during this initial phase.

Can this framework handle clients with multi-modal data? ...

Indeed, our framework is designed to be adaptable and can certainly accommodate scenarios where clients possess multi-modal data. While our primary focus in Modality Collaborated Federated Learning (MCFL) is on a more challenging setting where each client has access to only one modality, the framework is fully compatible with situations involving clients with multi-modal data.

In such instances, our approach involves sending modality-specific parameters for all pertinent modalities, in addition to the shared parameters, to the client for local training. This enables effective data-level collaboration across different modalities. This setup is particularly relevant and practical in scenarios like the healthcare sector, where it's common for a single client to handle multi-modal data. Our framework's flexibility in this aspect ensures its applicability and effectiveness in a diverse range of real-world settings.

Security and privacy are not discussed...

You're absolutely right in emphasizing the importance of security and privacy in federated learning, particularly within sensitive domains such as healthcare. In our paper, while the primary focus has been on the technical development of the Modality Collaborated Federated Learning (MCFL) framework, we acknowledge that security and privacy are critical considerations.

It's important to note that the implementation of our MCFL framework does not introduce any new risks beyond those already present in uni-modality federated learning. As such, the existing methods and best practices for ensuring security and privacy in traditional federated learning environments are applicable and effective in the context of MCFL.

We recognize the significance of addressing these issues explicitly and thoroughly. Therefore, we plan to expand upon security and privacy measures in future iterations of our work, ensuring a comprehensive and robust approach to these critical aspects.

We deeply appreciate your encouraging rating for our paper and are grateful for the support and guidance your feedback has provided. Your constructive comments have played a crucial role in helping us refine and improve various aspects of our work, enhancing its clarity, depth, and overall relevance. If you have further questions, we would like to provide a further explanation. Thank you for the invaluable insights and the confidence you have shown in our research.

As the interactive rebuttal window will close soon, we sincerely thank you for your valuable feedback. We hope that we have comprehensively addressed the concerns in our response and appreciate any additional feedback you may have. We appreciate your encouraging rating of our paper. Thank you again for your time.

Thank you for dedicating your time and expertise to review our paper. Your insightful feedback and the high rating are greatly appreciated, as they not only help refine our work but also motivate our continued research efforts. We are truly grateful for your valuable contributions.

The experiments are primarily conducted on two-modal settings, and the adaptability of FedCola to more complex scenarios with more modalities was not thoroughly studied. Could the proposed framework be extended to scenarios with more modalities?

Our current research primarily centers on vision and language modalities, but it's important to emphasize that our framework is compatible with scenarios encompassing additional modalities, a topic we explore in Section 5.4. Looking ahead, we are actively planning further studies to apply and refine our FedCola framework in more complex multi-modal environments. We are confident that this future work will further showcase FedCola's adaptability and robustness, expanding its applicability in diverse federated learning contexts.

The effectiveness of temporal modality arrangement was linked to the correlation between different modalities. Will the performance be influenced by the semantics of different modalities?

Indeed, the semantics of different modalities and their correlation play a significant role in the performance of our framework, as highlighted in our study. We delve into the impact of semantic correlation in Table 3 and provide an in-depth discussion on this subject in Section 5.3.

To illustrate, the correlation level between modalities influences the necessity of the heat-distribution stage in our framework. For example, CIFAR-100 and AGNEWS exhibit a relatively low correlation, implying that the shared parameters trained using the 'warming' modality are not readily adaptable to the 'warmed' modality. In such cases, freezing the weights during the heat-distribution stage can decelerate the adaptation process, rendering this stage less effective compared to scenarios involving modalities with higher semantic correlation, like OrganAMNIST and MTSamples. This aspect of our research underscores the importance of considering semantic correlations when applying our framework to different modality pairings.

Limited discussion on privacy issues: The consideration of privacy issues in federated learning is critical; however, the paper did not sufficiently address this issue in the MCFL.

We appreciate your concern about the treatment of privacy issues in our MCFL setting. We want to emphasize that MCFL does not introduce additional privacy risks beyond what is inherent in traditional federated learning setups. In MCFL, as is customary in standard federated learning, the clients' data remains local, with only the model updates being transmitted for aggregation purposes. This methodology ensures that the privacy of the data is preserved as safe as in traditional federated learning.

Our focus in this paper was primarily on addressing the technical challenges within the MCFL framework. However, we recognize the importance of privacy and aim to explore this in greater depth in future research.

The motivation of the modality compensation is based on an equation based on the number of training samples. However, in the figure provided, the misalignment is based on the number of sampled clients...

Thank you for highlighting this point. We acknowledge that there is an intrinsic relationship between the number of sampled clients and the number of training samples in our model. However, these two factors can be considered equivalent in our context, as clients are sampled uniformly across the board with a fixed ratio. Consequently, an increase in the number of sampled clients directly translates to a greater volume of training samples.

We have also taken this opportunity to correct a typo in the figure to more accurately depict the alignment. This amendment should provide a clearer understanding of how the number of sampled clients and training samples interact and influence the modality compensation in our framework.

Is the server model the same as the client models?...

As shown in Section 2 (Problem Definition) and Appendix E, the server model and the client models are different. The server model contains modality-specific parameters for all modalities ($\phi^{(m)}$) as well as the shared parameters ($\phi^{(m_s)}$), while the client model only contains the modality-specific parameters for its modality and the shared parameters. In this way, unrelated parameters don't need to be involved in the communication, which makes FedCola maintain the same level of communication as FedAVG.

We are grateful for the opportunity to refine our work based on your constructive feedback. Your comments have been instrumental in helping us enhance the clarity, depth, and relevance of our research. If you have further questions, we would like to provide a further explanation.

As the interactive rebuttal window will close soon, we sincerely thank you for your valuable feedback. We hope that we have comprehensively addressed the concerns in our response and appreciate any additional feedback you may have. Thank you again for your time.

We sincerely appreciate your thorough review and valuable insights, which have helped us identify areas in our research that require further clarification.

I have concerns about the motivation of this work. In this paper, each client has one single modality, which is not practical in the real-world setting. Though the authors use the hospital as an example, it is more practical that different clients may have different ratios of different modalities of data.

We appreciate your concerns regarding the practicality of our work. However, we believe that our setting is indeed practical and highly relevant in numerous scenarios involving sensitive data, particularly in the healthcare sector, as Reviwer RJk2 acknowledges our setting "more realistically reflects the nature of real-world data". In a hospital environment, for example, different departments (such as MRI and CT scan units) collect data in various modalities. Due to strict protocols, these departments often cannot freely share data, leading to a scenario where each department essentially acts as an independent client with uni-modal data within a federated learning system. This not only demonstrates the practicality of our approach but also highlights its necessity in certain contexts.

Moreover, we would like to clarify that our framework is designed to accommodate the scenario where each client has a single modality, but it is not restricted to this scenario alone. The modality-specific parameters in our framework can indeed encompass more than one modality. In cases where clients have data in different modalities, we can send the modality-specific parameters for all relevant modalities, along with shared parameters, for local training. The rest of the process remains the same, ensuring compatibility with the scenario you mentioned.

Additionally, our focus on scenarios where each client has access to only one modality is intentional, stemming from the challenge and its relevance to federated learning. When clients possess data in multiple modalities, direct data-level knowledge sharing is feasible, allowing the client model to be trained directly on multiple modalities. In contrast, when a client has access to only single-modality data, knowledge sharing must occur at the parameter level through the aggregation of parameters from other clients. This represents a more complex and challenging scenario. Meanwhile, it's worth noting that multi-modal learning within a single client aligns more closely with centralized learning paradigms. Our research, however, focuses on cross-modal learning through parameter sharing without direct data access, adhering to the core principles of federated learning.

Also, if I understand correctly, as stated in Sec. 5.2, the authors expect a better-aligned global model. However, assuming each client has one single modality, it should fall into the personalized federated learning domain, where we care more about the clients’ local performance.

We would like to clarify that personalized federated learning and traditional federated learning represent two distinct approaches with differing emphases. Personalized federated learning prioritizes the local performance of individual clients, whereas traditional federated learning is more concerned with the performance of the global model. Our work focuses on the global performance on each modality, which falls into the latter.

In the context of our work, while it's true that each client possesses data in a single modality, such a modality is not unique on one client but shared among multiple clients. Our primary objective isn't to further personalize the global model for each specific modality per client. Such personalization, if pursued, would be limited to a single modality, diverging from the central theme of our paper, which is modality collaboration. Our focus remains on enhancing the global model's performance through collaborative learning across different modalities, a concept that aligns with the overarching goals of traditional federated learning.

As the interactive rebuttal window will close soon, we sincerely thank you for your valuable feedback. We hope that we have comprehensively addressed the concerns in our response and appreciate any additional feedback you may have. We kindly request you to consider improving the rating of our paper. Thank you again for your time.

on methodology: Convolutional networks are perhaps a simpler type of models for vision related tasks. As in the CIFAR100 case, transformer-based models seem to be somehow below par of the performance of convolutional networks, I wonder if it is possible to incorporate convolutional networks in the study and see relevant results.

We respectfully disagree with the notion that transformer-based models are generally outperformed by convolutional networks in vision tasks. In recent years, transformer architectures have emerged as a leading choice in various vision-related applications, including image classification[1], object detection[2], and semantic segmentation[3]. Their ability to effectively process and interpret visual data has been well-documented in numerous studies. As in the CIFAR-100 case, Astroformer [4], which is in a transformer architecture, is the current state-of-the-art method according to the benchmark for image classification on CIFAR-100 without extra training data on paperswithcode.

Moreover, when it comes to multi-modal tasks, particularly those involving vision-language integration, transformers have become almost the de facto architecture[5,6]. This preference is largely due to transformers' superior capability in handling long-distance dependencies, a feature that is particularly valuable in text data and is not as effectively managed by convolutional networks.

Given that our research is primarily focused on multi-modal data within the context of federated learning, we have chosen to leverage the strengths of transformer models. Their adeptness at handling multi-modal data makes them particularly suitable for our goal of facilitating modality collaboration in a federated learning setting. As such, our study concentrates on the use of transformer models, and incorporating convolutional networks into MCFL falls outside the current scope of our research.

[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2020

[2] DETRs with Collaborative Hybrid Assignments Training, 2023

[3] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks, 2022

[4] Astroformer: More Data Might not be all you need for Classification, 2023

[5] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models 2023

[6] VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts, 2021

We hope that these responses will address your concerns and offer additional clarity regarding our research. We appreciate the opportunity to engage in this constructive dialogue and would be grateful if you could consider improving the rating of our paper in light of these clarifications. If you have further questions, we would like to provide a further explanation. Thank you for your time and thoughtful feedback.

Before delving into the specifics of your queries, we would like to extend our sincere gratitude for the time and effort you have invested in reviewing our submission.

• on methodology: what is the frequency of doing inter-modality update of the shared parameters and that of doing intra-modality update of modality-specific parameters?
• on methodology: as the number of training data points/clients in different modalities could be different, the shared parameters should be far more frequently updated compared to modality-specific parameters. It seems that authors have proposed the aggregation step to address this problem. It remains unclear to me how gradient back-propagation works with respect to those parameters which are manually aggregated/augmented into a model, ...

Your questions are raised due to confusion regarding the underlying federated learning (FL) process of our setting. To clarify, the frequency of updates for all parameters, whether they are shared or modality-specific, is once per round, so the shared parameters are not more frequently updated. Such updating in most federated learning frameworks is not directly via back-propagation but via an aggregation of all client models. However, this process can be conceptually understood as an equivalent operation to performing back-propagation on the global model at the server level. Let us provide a formal demonstration of how back-propagation equivalently operates for the global model in FL, and we've added it to Appendix A in the revised version (colored orange):

Consider the aggregation process on the server for a parameter $w$ from $N$ client models $w_i$, with each client having $m_i$ training samples. The update of $w$ at round $t$ using the FedAVG algorithm can be expressed as:

where $p_i = \frac{m_i}{m}$ and $m = \sum_i m_i$. Here, $m_i$ represents the number of training samples on client $i$.

Given that both the global and client models start with the same initialization $w^{(t-1)}$, the updates on client $i$ can be formulated as

using gradient descent for $E$ epochs with a learning rate $\eta$ and the local objective $F_i$.

The aggregation can then be further expressed as:

with

Thus, it results in one step of update per round for all parameters. The primary distinction between modality-specific and shared parameters lies in the number of clients where the gradient is averaged: modality-specific parameters receive a weighted average of gradients from clients with the corresponding modality, whereas shared parameters are averaged from all clients. As we discussed in Section 5.2, such a distinction may lead to misaligned generalizability among all layers. Furthermore, we address this distinction on total client numbers, instead of frequency, with our proposed modality compensation scheme, ensuring that both shared and modality-specific parameters are updated effectively to suit their respective roles in the federated learning process.

..., and how these external parameters function during the inference process of a certain modality?

For the inference process of a certain modality, only the modality-specific parameters for that modality as well as the shared parameters will be involved in the inference process, whereas the other parameters for external modalities won't be involved.

Authors rebuttal response did not address my main concerns on the methodology as well as the conclusiveness of presented numerical evidence. Based on authors’ rebuttal reply, I find the reported results more dubious. Therefore, I will keep my score as rejection.

(1) on uni-modal baseline accuracy: More clarify is needed about the method Vanilla modality-agnostic transformer and its implementation. The reported benchmark in table 1 that sharing all weights destructs the functionality for the image application in entirety and meanwhile slightly boosts the performance in the text application is highly counter-intuitive and suspicious, as the image and text dataset are utterly irrelevant. In multi-modality learning, in order to benefit one domain by leveraging information from the other domain, the information needs to be well-coordinated. See reference [1,2,3,4,5,6].

Given that the two datasets (CIFAR100 and AG_News; MedMNIST v2 (using the name in the quoted reference in Yang et al., 2023) and MTSamples) used in the experiments are irrelevant, it does not make sense that a network leveraging information from another domain is going to perform better than the model trained in the uni-modal setting with proper finetuning.

[1] A Deep Multi-Modal CNN for Multi-Instance Multi-Label Image Classification, Song et al, 2018

[2] Multimodal deep networks for text and image-based document classification, Audebert et al., 2019

[3] Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers, Huang et al., 2020

[4] Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking, Wang et al., 2019

[5] Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video, Li et al., 2020

[6] Dynamic Modality Interaction Modeling for Image-Text Retrieval, Qu et al., 2021

(2) on averaging of shared parameters: Authors make it very clear that there is no explicit gradient-based update that transpires during the weight sharing process (quote Rebuttal 1/3). This has erased any consideration of potential regularization effect in the optimization process, and makes it even more unlikely to let completely irrelevant information from one domain to benefit another.

(3) on accuracy characterization regarding each modality: It does not make sense to compute mean across modalities, when the task in each domain is completely irrelevant to the other. To characterize optimization performance with multiple objectives, conventional characterization methods such as Pareto front [7] should be used.

[7] Multiobjective Optimization: Interactive and Evolutionary Approaches, Lecture Notes in Computer Science, Zitzler et al., 2008

[8] Convex Optimization, Vandenberghe and Boyd, 2004

(4) on disproportionate data size across modalities and different number of agents across modalities

a. Quoting authors’ reply (2/3), the result on the dataset combination (MedMNIST and MTSamples) is far from what was reported in table 4 (image classification accuracy greater than 90%).

b. The AGNews has twice as much data compared to CIFAR100. When the number of clients in the data-abundant text domain is also larger than its counterpart (16 to 4), by the design of the aggregation in (3), the attention weights from the image domain with less data points will become irrelevant after aggregation, and the resulted performance should thus gravitate towards the scenario reported in table 1 (88.13% text classification accuracy in non-sharing). This contradicts authors’ reply (2/3).

I thank authors’ effort of presenting detailed response, and acknowledge authors’ point that attaining over 50% accuracy on CIFAR100 is of reasonable performances within large transformer models, FedAvg training algorithm and Dirichlet-distributed training data across clients.

As the interactive rebuttal window will close soon, we sincerely thank you for your valuable feedback. We hope that we have comprehensively addressed the concerns in our response and appreciate any additional feedback you may have. We kindly request you to consider improving the rating of our paper. Thank you again for your time.

(2) on averaging shared parameters:

As I pointed out earlier, there is no regularisation effect of doing such an averaging for the optimisation process. Considering that directly averaging weights give dubious numerical results when the data in two domains are unpaired and irrelevant, it does not make sense for the weight averaging to force a multi-modal benefit.

(3) meaningfulness of the result presentation:

Computing the arithmetic mean of the accuracy results across different modalities does not make sense, as the number of clients, amount of data and training setup in each modality can vary drastically.

Authors should report the accuracy in each modality under various training settings clearly, and this will take more than a major revision to complete.

(4) contradiction between the rebuttal information and what was reported in the main text:

My question about the contradiction in reported results is not answered directly or explicitly (point 4b in previous reply).

The supplementary table in rebuttal (2/3) concerns $\alpha=0.1$ and $r=0.25$, as marked in orange in the revised manuscript. This combination did not appear in the main text Table 4 or Table 1, which makes the result lack a proper reference. Authors did not specify results per each modality in the supplementary table in the supplementary table either.

Considering that authors are experimenting with less than 20 clients in total and the amount of data per client is rather sufficient, the uni-modal accuracy result, for text task or for image task, should not vary significantly given the data distribution and ratio of client participation are the same, regardless of total number of clients.

As I pointed out in my previous comment, the result in the supplementary table and the result in the main text are inconsistent.

In conclusion, there is reasonable doubt about the correctness of the setup of this work and the numerical result reported herein.

(1) Regarding the meaningfulness of the multi-modal setup in this work:

a. The CIFAR100 (Krizhevsky et al., 2009) and AG_News (Zhang et al., 2015) datasets are irrelevant, simply by checking their corresponding definitions. Such training setup renders the entire training task to be untrustworthy, particularly the claim in Table 1 that by leveraging information from AG_News, a network that only has 3% accuracy on CIFAR100 can get boosted to 56% accuracy.

With authors’ multiple round of reply, it becomes clearer that the multi-modality setup (training paradigm and result comparison) in this work is incorrect.

As I stated in my previous comments, the dataset used for multi-modality needs to be meaningfully paired/manually annotated, for instance Flickr30k dataset[cite1], Microsoft COCO [cite2], Visual Genome [cite3], SBU Caption dataset [cite4], Conceptual Captions[cite5], and Conceptual 12M[cite6]. All these datasets for multi-modality study are public and are actually used in the six references that authors cited earlier in their first round rebuttal reply.

[cite1] From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions, Young et al., 2014

[cite2] Microsoft COCO: Common Objects in Context, Lin et al., 2015

[cite3] Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, Krishna et al., 2017

[cite4] Im2Text: Describing Images Using 1 Million Captioned Photographs, Ordonez et al, 2011

[cite5] Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning, Sharma et al., 2018

[cite6] Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts, Changpinyo et al., 2021

b. The two references authors mentioned do not support their claim either.

-- In [r1], the used PAIRED image-text multi-modal datasets are Microsoft COCO and Flickr30k, respectively. It has been made clear in the original text that “we distribute ... CIFAR100 to 10 unimodal image clients, and AGNEWS to 10 unimodal text clients” and that “All unimodal algorithms are extended to multimodal scenarios by operating on each modality separately, e.g., FedGEMS performs per-modality local distillation and entropy-based selective aggregation.” This is similar to [cite7] and [cite8] where knowledge distillation is done to extract information within each modality with unlabeled data.

In other words, there is categorically no interaction between information of CIFAR100 and information of AG_News in the work [r1] cited by authors.

Also, in [r1], only result in the image modality is reported.

[cite7] DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, Sanh et al., 2020

[cite8] Ensemble Distillation for Robust Model Fusion in Federated Learning, Lin et al,, 2020

-- Authors of [r2] manually created 5 tasks and each task only concerns one modality. In [r2], every task in each modality is still trained separately and their loss functions in the training process are manually added together. There is no interaction between information of CIFAR100 and information of AG_News.

c. authors have not yet directly addressed how the pairing between MedMNIST and MTSamples are meaningfully generated for the multi-modal study.

(1) The major concern on the main method of this submission is that it does not make sense to manually pull two utterly irrelevant datasets, conduct jointly training, directly add up model weights and take average and simply claim that the training results are better because of using more data.

• In the federated setting, authors can take a reference with [FedMultimodal: A Benchmark For Multimodal Federated Learning, Feng et al., KDD 2023] to explore commonly used multi-modal dataset and present a clear argument about the comparison between the fusion scheme presented therein and the proposed method.
• Also, [A unified framework for multi-modal federated learning, Xiong et al., Neurocomputing, 2022] is another explicit example where a weight matrix is used to correlate feature from different modalities. There needs to be a non-trivial action leveraging inter-modality information.

(2) Methodology for generalization

• In my previous reply, I have already given specific examples of multi-modal learning where I expressed the point that inter-modality tasks need to be related.
• Authors have emphasized on parameter sharing, while throughout the main text there is no explicit discussion about the difference between proposed method and some established multi-task learning or feature alignment work. Insufficient discussion with prior related work thus incurs the need for major revision of the current manuscript.
• Extant methods for improving generalization that are similar to what was suggested in this work are representation learning (domain generalization/adaptation, feature alignment), multi-task learning or transfer learning. In these established directions of handling information across multiple domains/modalities, involved tasks/domains are either information-wise relevant or sharing certain overlap of data. I give examples below to demonstrate specific measures to leverage inter-domain information:

  • Domain generalization:

    • Kernel method is used to explicitly correlate features from different domains [Blan21]
    • Multi-domain MMD-based regularization: a Maximum Mean Discrepancy penalty between feature distributions is added to loss for learning feature representations across different domains [Li2018]

  • Multi-task learning:

    • For hard parameter sharing, Caruana [CAR98] made clear that “uses the training signals for related tasks … to improve generalization” and so are the examples (steering directions and traffic condition determination) given therein.
    • For soft parameter sharing, Duong et al.[Duong15] use Frobenius norm based penalty and Yang et al. [Yang17] use trace norm based penalty to regulate parameters in different tasks.

[Li2018] Domain Generalization with Adversarial Feature Learning, Li et al., 2018 CVPR

[Car98] Multitask Learning, R. Caruana, 1997 Machine Learning Springer

[Duong15] Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser, Duong et al., 2015, ACL

[Yang17] Trace Norm Regularised Deep Multi-Task Learning, Yang and Hospedales, 2017 ICLR

[Blan21] Domain Generalization by Marginal Transfer Learning, Blanchard et al., 2021 JMLR

(3) Inconsistency in reported results:

• Exploring the varying number of clients and presenting comparable results with content in the main manuscript is a comment in my original comment (dated Oct 31, 2023). Authors had the entire rebuttal period to prepare comparable $(N_l, N_v)$ accuracies with the content in the main text ($\alpha$,$r$ setup), but did not present such an update. Claiming lack of time is an invalid excuse.

(4) Dataset choice for experiments:

Authors fail to justify the selected two pairs of text-image datasets. Both datasets have completely irrelevant text and image components, which violates the current common practice in multi-modal learning.