Task-Oriented Multi-View Representation Learning

Thank you for your constructive comments.

Response to Weakness 1 (Experimental settings):

We apologize for some distress caused by the unclear experimental setup in the submitted manuscript. It should be clarified that our experiments are fair because all methods (including our TOMRL) use the same data in the training and testing stages. Specifically, both the comparison methods and baseline methods were allowed to train to convergence on the full dataset $D$. We then constructed $D$ as a set of (multi-view few-shot) tasks and randomly selected 100 tasks for testing. The remaining tasks were used for the training of the meta-learner in an episodic fashion, as shown in Alg. 1 in the manuscript. The well-trained TOMRL was also tested on the same 100 tasks. Note that these test tasks are never used to fine-tune any method to ensure a fair comparison. In other words, they are new and unseen tasks for both the compared methods and our TOMRL. This experimental setting ensures that the performance gains from TOMRL are derived from meta-learned task-level knowledge.

Response to Weakness 2 (Why TOMRL works):

As mentioned in the response to Weakness 1, we train the meta-learner additionally at the task level for the well-trained existing multi-view representation model. Indeed, the meta-learner is expected to learn a set of mappings from task embeddings to modulation parameters, in order to quickly adapt to unseen new tasks. More specifically, for each task in the meta-training stage, the meta-learner generates modulation parameters for the support set and performs clustering or classification on the query set using the modulated features. Minimizing the loss on the query set can induce the meta-learner to learn better mappings.

Response to Weaknesses 3 (Additional experimental results):

As a supplement, the following tables list the results of the TOMRL method on the four datasets suggested by the reviewer. Note that the results on the PatchedMNIST dataset are not included because PatchedMNIST is a subset of the MNIST dataset containing only the first three classes [1], which does not satisfy the settings of the 5-way task.

Table: The ACC$_{clu}$ values on 100 5-way 5-shot clustering tasks

Table: The NMI values on 100 5-way 5-shot clustering tasks

[1] Trosten, Daniel J., et al. On the Effects of Self-supervision and Contrastive Alignment in Deep Multi-view Clustering. In CVPR 2023.

Response to Questions (Minor error):

Thanks for pointing out the error. We've fixed this error in the revised manuscript: switched lines 20 and 21 in pseudo-code.

Thank you for your constructive comments, especially on the multi-view representation learning framework, which is helpful to improve our work!

Response on Motivation and Innovation (Weaknesses 1 & 3 & 4):

To more clearly demonstrate our motivations and innovations, we review related research, especially in multi-view multi-task learning (MVMTL) and continuous multi-view learning (CMVL), which both involve multi-view and multi-task scenarios.

• MVMTL follows the multi-task learning setting, where the learning objective is to improve performance on the main task through joint training on multiple related (auxiliary) tasks. The counterpart is conventional (single-view) multi-task learning, with the difference that MVMTL utilizes comprehensive information (e.g., consistency and complementarity, etc.) from multiple views of each task. Once the multi-view information is well integrated, most MVMTL methods degrade to the conventional multi-task scenario [1].
• CMVL follows the continuous learning setting, where the learning objective is to handle new tasks without performance degradation on previously learned tasks for a series of consecutive tasks. Similarly, the counterpart is conventional (single-view) continuous learning, with the difference that CMVL utilizes comprehensive information from multiple views of each task.
• Unlike the above two types of methods, our TOMRL aims to learn the task-oriented multi-view representation with the goal of learning task-level experience from numerous historical multi-view tasks, enabling the model to adapt quickly when dealing with unseen new multi-view tasks.

MOTIVATION: Existing models for learning multi-view representations (especially in unsupervised settings) mostly focus on how to integrate comprehensive information from multiple views at the data level while neglecting fast adaptation on different tasks. INNOVATION: We provide a meta-learning perspective for task-oriented multi-view representation learning. 1) We define the ``episode'' form of supervised and unsupervised multi-view learning tasks, and construct meta-training and testing sets at the task level. 2) Based on the two basic principles mentioned in Section 3, we design a meta-learner for feature modulation and train it in an episodic fashion. 3) For an unseen new multi-view task, the well-trained meta-learner generates a set of modulation parameters that help the model quickly derive the uniform representation specific to this task.

Response on multi-view representation learning framework (Weakness 2):

The multi-view learning framework mentioned in this paper is extended by the deep multi-view clustering framework [2]. Thanks to your valuable comments, we rethink the rigor of this extension. Following the survey studies[3, 4], multi-view representation learning can be roughly divided into alignment-based and fusion-based methods. Let $x^1$ and $x^2$ denote the two views of sample $x$, alignment-based methods can be symbolized as $f(x^1;W_f)\leftrightarrow g(x^1;W_g)$, where $f$ and $g$ denote the feature extractors of view 1 and view 2 with corresponding parameters $W_f$ and $W_g$, respectively, and $\leftrightarrow$ denotes the alignment operation. Similarly, the fusion-based approach can be expressed as $H=F((f(x^1;W_f), g(x^1;W_g)); W_F)$, where $F$ is the fusion operation with the parameter $W_F$, and $H$ denotes the fused unfied representation. Although recent work [5] has attempted to integrate the two into a unified framework, it is clear that our TOMRL is strictly more suitable for fusion-based multi-view representation learning methods due to the fusion operations involved. Therefore, we modify ''the multi-view representation learning framework'' to ''the fusion-based multi-view representation learning framework'' and clarify the research scope in the revised version.

[1] Lu, Run-kun, et al. Multi-view representation learning in multi-task scene. Neural Computing and Applications 32 (2020): 10403-10422.

[2] Trosten, Daniel J., et al. On the Effects of Self-supervision and Contrastive Alignment in Deep Multi-view Clustering. In CVPR 2023.

[3] Li, Yingming, et al. A survey of multi-view representation learning. IEEE TKDE 31.10 (2018): 1863-1883.

[4] Yan, Xiaoqiang, et al. Deep multi-view learning methods: A review. Neurocomputing 448 (2021): 106-129.

[5] Wang, Ren, et al. MetaViewer: Towards A Unified Multi-View Representation. In CVPR 2023.

Thank you for your constructive comments.

Response to Weakness 1 (Different dataset domain):

Thank you for your valuable suggestion on ''experimenting on different dataset domains’’. The performance of TOMRL in cross-domain scenarios has been validated on two variant datasets, NoisyFashion and EdgeFashion, in Sec. 5.3 of the manuscript. The experiment setting of ``cross-domain between NoisyFashion (2 views) and Caltech101-7 (6 views)'', mentioned by the reviewer, is challenging for the current framework. This challenge comes from the fact that existing multi-view models are sensitive to the number of views. Although the gradient-based task embedding (in Sec. 3.1.2) provides flexibility in handling tasks with different scales (i.e., ways and shots), TOMRL, and in particular the meta-learner, is still based on the existing multi-view models. Therefore, the key to implementing the cross-domain "between NoisyFashion and Caltech101-7" is to design a unified model that can handle data with different numbers of views, which is not the focus of this work but is worth investigating and is part of our future work.

Response to Weakness 2:

Thank you for your valuable suggestions. The revised manuscript explains the special markings in each table title.

Response to Questions:

We rechecked the experimental results. One possible reason is that TOMRL improves the number of samples clustered correctly, but does not significantly improve the cluster distribution obtained by InfoDDC in this scenario.

Thank you for your constructive comments, which were inspirational in refining the manuscript.

Response to Weaknesses 1 & 3 (Connections and differences with multi-view multi-task learning (MVMTL) and continuous/lifelong multi-view learning (CMVL)):

MVMTL, CMVL, and our TOMRL all consider both multi-view and multi-task scenarios, exploiting comprehensive feature representation of multiple views in each task as well as the task relationships of multiple related tasks. However, they are different in terms of learning objectives, key challenges, and practical settings.

• MVMTL follows the multi-task learning setting, where the learning objective is to improve performance on the main task through joint training on multiple related (auxiliary) tasks. Key challenges include selecting/designing auxiliary tasks, balancing loss functions across multiple tasks (Weakness 4), etc. The counterpart is conventional (single-view) multi-task learning, with the difference that MVMTL utilizes comprehensive information (e.g., consistency and complementarity, etc.) from multiple views of each task. Once the multi-view information is well integrated, most MVMTL methods degrade to the conventional multi-task scenario [1].
• CMVL follows the continuous learning setting, where the learning objective is to handle new tasks without performance degradation on previously learned tasks for a series of consecutive tasks. The key challenge is to overcome catastrophic forgetting. Similarly, the counterpart is conventional (single-view) continuous learning, with the difference that CMVL utilizes comprehensive information from multiple views of each task.
• Our TOMRL follows the meta-learning setting, where the learning objective is to rapidly handle new downstream tasks by learning meta-knowledge over multiple tasks. To this end, we construct task-level training and test sets on the multi-view dataset and train the model in an episodic fashion. The counterpart is conventional (single-view) meta-learning, with the difference that TOMRL considers both view-specific representations and fused unified representations in the meta-learning process. If the multi-view properties of the task are ignored, TOMRL degrades to a regular meta-learning approach.

In addition, we review existing works involving ''meta-learning'' and ''multi-view learning'' in the manuscript, and most of them aim to improve the performance of meta-learning methods by utilizing multi-view information rather than the fast adaptation of multi-view representations in different downstream tasks.

[1] Lu, Run-kun, et al. Multi-view representation learning in multi-task scene. Neural Computing and Applications 32 (2020): 10403-10422.

Response to Weakness 2 (Feature alignment of similar instances in different tasks):

We apologize for some misunderstandings due to unclear writing in the manuscript. Firstly, the basic principle ‘’b) the fusion process of features may be inconsistent" mentioned in Section 3 means that ‘’given multiple view-specific features, the unified representation fused for different downstream tasks should be different using the same fusion strategy’’, rather than ‘’the fusion strategy being performed in different tasks should be different’’. In contrast to the latter, the former does not suffer from feature misalignment due to the same fusion strategy. Existing multi-view representation learning methods pay little attention to this unified representation inconsistency, i.e., once the feature extraction and fusion strategies are given, the unified representation is determined. Part of our motivation comes from this gap between existing methods and the basic principle (the other part comes from the basic principle (a) in Section 3). Secondly, our TOMRL inherits model-agnostic advantages from gradient-based meta-learning, which makes it flexible enough to be integrated into a variety of multi-view learning models and, of course, a variety of fusion strategies.

Response to Weakness 4 (Weights of the loss function):

In the response to Weakness 1, we discussed the differences between the multi-view multi-task learning methods and the proposed work. The weighting of the loss function is a key issue in jointly training multiple tasks, but is not a concern in TOMRL, which follows a meta-learning "episode" training fashion.

Response to Weakness 5 (Notation mistakes):

Thank you very much for your constructive suggestions. These notation mistakes have been revised in the revised manuscript.