Multimodal Meta-learning of Implicit Neural Representations with Iterative Adaptation

Q1: Previous research [1-4] has explored cross-modal relationships in multimodal data extensively. Some even utilize Transformer structures for multimodal meta-learning.

A1: We kindly remind the reviewer that our problem focus is on employing a meta-learning approach to better approximate joint multimodal continuous functions. To the best of our knowledge, among existing multimodal meta-learning research, Multitask Neural Processes (MTNPs) is the sole study focusing on a problem context similar to ours. Importantly, as noted by the reviewers gNAu and cLC2, our work stands out as novel within this domain, proposing an interesting idea to accelerate the convergence of independent unimodal learners by exploring the multimodal structures in their on-going learning states during iterative adaptation procedures.

Besides MTNPs, existing multimodal meta-learning studies [1,2,3,4] mentioned by the reviewer differ significantly from our work in terms of the definition of modality and problem scenarios, making direct comparisons challenging and unfair.

Vuorio et al. [1, 3] and Abdollahzadeh et al. [2] address image classification on a union of digit, bird, aircraft datasets, where each dataset structures the unique modality. This notion of multimodality should not be confused with the conventional understanding of multimodality in data types (e.g. images and texts), as emphasized in Section 3 in Abdollahzadeh et al. [2]. Due to this disparity in the notion of multimodality, their approaches inherently reduce to unimodal meta-learning methods in the context of our joint multimodal signal modeling.

On the other hand, Sun et al. [4] deals with many-to-one multimodal classification problems using jointly sampled images, audios, and texts. Their proposed meta-learning approach involves the independent adaptation of each modality-specific encoder within the inner loop, followed by their fusion in the outer loop. Here, the transformers are used for each of the unimodal encoders, not for multimodal fusion. This is unlike ours that explores the joint iterative adaptation of independent learners in the inner loop via SFTs, which prevents overfitting of individual learners and accelerates their convergence during the inner-loop adaptation process.

(response continued in the following thread)

Q1: This loss function is a straightforward L2 loss, which is not inherently problematic but may not be the best choice for all types of problems. For instance, if the goal is robustness against outliers, then an L1 loss or Huber loss might be more suitable. The paper does not discuss the choice of loss function and its suitability for the tasks at hand.

A1: We opt for L2 (MSE) loss since we find it is the most common choice for the loss function in the literature of implicit neural representations. Nevertheless, we agree with the reviewers that the optimal choice for the loss function could vary for each problem, and leave this investigation as our future work.

Q2: Comment: Equation 2 extends Equation 1 by adding context parameters ϕ. However, the paper does not provide a mathematical justification for the choice of this extension. Weakness: The addition of context parameters ϕ increases the model complexity without a detailed explanation or justification. This can be an issue if the goal is to keep the model as simple as possible for interpretability or computational efficiency.

A2: Our extension from Equation 1 to Equations 2, 3, and 4 relies directly on CAVIA; such a formulation is extensively studied in various domains, including implicit neural representations.

It's important to note that the introduction of context parameters ϕ of CAVIA offers several advantages over MAML in terms of overfitting, computational efficiency, and interpretability. For instance, by separating task-specific adaptations (ϕ) from task-agnostic generalizable features (θ), CAVIA effectively mitigates overfitting and reduces computational overhead during adaptation. In addition, the independent adaptation of ϕ allows for efficient parallelization, thereby reducing the time required for meta-learning. Moreover, the context parameters (ϕ) are found to capture the latent task structure and operate as low-dimensional task-specific embeddings, leading to greater interpretability than MAML. We kindly refer the reviewer to the original paper of CAVIA for the detailed discussion.

Q3: Comment: The meta-objective in Equation 3 and 4 is a standard formulation. However, the paper does not discuss the potential issues that could arise from a bi-level optimization problem, such as saddle points or local minima. Weakness: The paper lacks a rigorous mathematical analysis of the optimization landscape, which is crucial for understanding the method's efficiency and effectiveness.

A3: We acknowledge the reviewer’s concern. However, as mentioned by the reviewer, such a bi-level optimization in Equation 3 and 4 is standard formulation in the widely studied model-agnostic meta-learning literature. Moreover, none of the optimization issues was encountered in any of the experiments conducted for our paper. Please note that such potential pathologies can be readily alleviated by utilizing gaussian-blurred smooth objectives [ES,PES], and we leave this investigation as part of our future work.

[ES]: Luke Metz, Niru Maheswaranathan, Jeremy Nixon, C. Daniel Freeman, Jascha Sohl-Dickstein, Understanding and correcting pathologies in the training of learned optimizers, In ICML, 2019.
[PES]: Paul Vicol, Luke Metz, Jascha Sohl-Dickstein, Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies, In ICML, 2021

Q4: Comment: These equations introduce an elaborate framework that involves several novel components, like State Fusion Transformers (SFTs). However, it's not clear how these equations were derived or why they are theoretically sound. Weakness: The paper introduces several novel ideas but does not provide a theoretical justification for them. This lack of theoretical grounding makes it difficult to assess the quality and applicability of the proposed method.

A4: Equations from 5 to 7 are equivalent to meta-learning each unimodal framework separately. In addition, Equations from 8 to 11 for MIA share a similar spirit with the studies in the learning-to-optimize domain, where our MIA can be interpreted as an extension of such widely studied learned optimization algorithms to approximating joint multimodal signals, exploring an interesting idea to accelerate the convergence of independent unimodal learners by capturing the multimodal structures in their on-going learning states and landscapes during iterative adaptation procedures.

Q12: The manuscript mentions that the audio signals were trimmed. However, it does not explain how this preprocessing step might affect the meta-learning process. Could you explain how the trimming of audio signals might have affected the results and why this preprocessing was necessary?

A12: In our study, we performed two preprocessing steps on the audio signals. First, we (1) decreased the sampling rate of the audio signals, followed by (2) standardizing all audio signals to a consistent length through trimming or zero-padding. Step (2) is a common practice in audio-related domains, facilitating batching during training and evaluation, which significantly reduces computational time.

The decision for step (1) was specifically driven by the complexities encountered in training the multimodal baseline model, MTNPs. MTNPs exhibit quadratic complexity with respect to the support/query set size, due to the attention mechanisms directly applied to coordinate-feature pairs within these sets. This imposes excessive demands on memory and computational resources, particularly when handling audio signals, making the training process infeasible without decreasing the sampling rate.

During the preprocessing of these audio signals, we confirmed that the reduction in sampling rate did not significantly simplify the problem, ensuring that the semantic integrity or quality of the original sounds remains still. Instead, it notably reduced the redundancy within the audio data, making the training of MTNPs considerably more feasible.

Q13: While the section provides a valuable ablation study to understand the impact of various modules, it is relatively shallow. For instance, it would be beneficial to understand how each of these modules contributes to reducing overfitting or improving convergence speed.

A13: We appreciate the constructive feedback. In response, we conducted a more in-depth ablation study to include all possible component combinations to analyze how each of them contribute to performance. Due to time and computational limitations, we focused on the impact of each component on vanilla Composers in the synthetic dataset. The results are below.

By comparing the results (1) and (2), we find that FusionMLPs only can greatly enhance the gradients of vanilla Composers, such as by modifying the gradient direction or magnitude, or even the ones in combination with USFTs, MSFTs or both (3 vs 4, 5 vs 6, 7 vs 8). The improvement in memorization capability is more significant when vanilla Composers is further augmented with USFTs (1,2 vs 3,4), which is aligned with the study in our main paper. This indicates that USFTs is especially beneficial for capturing modality-specific patterns in the states of the learners. Unlike USFTs, we observe that MSFTs excel in utilizing cross-modal interactions among the learners’ states (3,4 vs 5,6,7,8), boosting the generalization performances of Composers significantly. Lastly, the most optimal performance is achieved when utilizing a combination of USFTs, MSFTs, and FusionMLPs, underlining the indispensable roles of each component within SFTs. This result validates the unique and crucial contribution of each component to the overall effectiveness of Composers.

Q14: The section uses relative error reduction as a metric but does not justify why this is an appropriate measure of performance. It might be valuable to consider other metrics like F1-score or ROC AUC, especially when comparing across multiple modalities.

A14: We thank the suggestion. We didn't explore the use of F1-score or ROC AUC since these metrics are designed for assessing (binary) classifiers, particularly in situations where classes are imbalanced.

As our paper primarily focuses on regression problems on joint multimodal continuous functions, we opted for alternative evaluation metrics, i.e. the average relative error reduction across sampling ratio ranges and modalities. The reasons are two-folded: (1) it remains robust against widely varying MSE magnitudes or scales influenced by modalities and sampling ratios, and (2) it offers a holistic understanding of the individual components' impacts within our ablation study.

It's worth noting that Kim et al. also employed the same metric (referred to as relative performance gain in their paper) for similar reasons and purposes. They utilized this metric in their investigation of MNTPs' capability to capture cross-modal relationships (please refer to Section 5.3 and Table 4 in Kim et al.).

Reference: Donggyun Kim, Seongwoong Cho, Wonkwang Lee, and Seunghoon Hong. Multi-task neural processes. In ICLR, 2022.

Q9: The manuscript does not delve into the qualitative implications of the MSE values reported. How do these numerical metrics translate into practical improvements? For instance, do lower MSEs correlate with visibly better reconstructions in real-world applications?

A9: Lower MSE values indeed signify reduced error in signal reconstruction or prediction. As pointed out by the reviewer, however, these numerical improvements might not perfectly align with human perception or subjective judgment of signal quality. Nevertheless, we opt for it as our performance measure since MSE and its logarithmic derivative PSNR are commonly used metrics for evaluating the fidelity of reconstructions in implicit neural representations learning domains.

Q10: Given the critical nature of climate data, an analysis of how errors in the model's predictions could propagate into real-world applications would be valuable.

A10: We believe understanding and mitigating such errors are fundamental to ensuring the reliability and practicality of any machine learning framework, and thereby we value the reviewer's insightful suggestion. Inaccurate climate predictions have far-reaching implications, potentially impacting (1) the formulation of environmental policies and disaster management strategies, (2) industries and economic planning reliant on climate forecasts, and (3) overall societal safety and preparedness.

Q11: The section does not sufficiently justify the choice of the AVMNIST dataset. Given its unique challenges, why was it selected over other multimodal datasets? It's mentioned that MTNPs fail at approximating the audio signals properly. A deeper analysis into why this failure occurs could offer valuable insights into the limitations of existing methods, thereby contextualizing the contributions of the proposed method more effectively.

A11: The rationale behind incorporating the AVMNIST dataset in our experiments is to showcase our approach's efficacy in handling diverse joint multimodal function modeling scenarios. Unlike datasets such as synthetic, CelebA, and climate data that commonly exhibit strong axis-aligned relationships among functions, real-world scenarios often lack such alignment. For instance, audiovisual signals present distinct function domains (or coordinate systems) between image and audio signals, posing a significant challenge for models in capturing cross-modal relationships due to the absence of explicit spatiotemporal correspondence.

Regarding the failure of MTNPs in approximating audio signals within the AVMNIST dataset, the MTNPs’ architecture relies on an "across-task inference" mechanism that facilitates cross-modal information exchange among axis-aligned features (see Equation 13 or Figure 2 in the corresponding paper). In AVMNIST, however, we omitted this mechanism due to the lack of explicit coordinate correspondence between images and audios, which was inevitable to train and evaluate MTNPs in AVMNIST. It’s noteworthy that cross-modal interaction can still be captured in the latent path of the first stream, as described in the Appendix D.2.

To validate the above discussion further, we conducted additional experiments investigating MTNPs' dependence on axis-aligned attention mechanisms. Specifically, we compared MTNPs' performances when trained with or without such mechanisms in axis-aligned multimodal CelebA datasets. We present the results in the table below. The results indicate that MTNPs without axis-aligned attention significantly underperformed. This suggests the essential role of axis-aligned attention in MTNPs for approximating multimodal functions, indicating potential limitations in modeling heterogeneous multimodal functions that cannot use axis-aligned attention effectively.

Q1: Why MIA can better handle limited data

A1: When data of one modality is limited, the computed gradients from the learner of that modality become inevitably noisy, which slows down the convergence of the learner and triggers overfitting. Interestingly, we find that our SFTs, through attention mechanisms, can incur positive knowledge transfer from the learner whose data is sufficient to the other learners with the limited data. This positive transfer compensates for low-quality gradients, enabling enhanced guidance to the learners. Moreover, SFTs prevent negative transfer as well, refraining from updating the state representations of the learner for a specific modality when it observed sufficient data. We included the detailed discussion on this analysis in Section 5.5 (marked in blue) and Appendix F.1 of the paper.

Q2: Why do we need to apply multimodal adaptation sequentially or iteratively?

A2: Since the total amount of observable data is fixed during the adaptation, any attempt to update each learner with its computed naive gradients has potential to hinder the convergence or trigger overfitting, particularly when data is limited. Therefore, our meta-learning framework involves multimodal adaptation throughout all optimization steps.

To validate this claim, we conducted additional experiments, wherein we compared our MIA with the two alternative ablative approaches: applying the multimodal adaptation only at the first (MA-First) or last (MA-Last) optimization step. The experimental results on the synthetic dataset are below.

As observed, MA-First consistently exhibits poor generalization (R <= 0.05), suggesting that the subsequent independent adaptation of each learner with limited data triggers overfitting. Conversely, MA-Last's late fusion strategy appears to address overfitting when data is scarce (R <= 0.05), evidenced by its improved generalization performances. However, as a notable downside, this fusion strategy tends to disrupt previously good solutions obtained with sufficient data, resulting in poorer memorization performances (R >= 0.05).

Crucially, MIA consistently outperforms these single-time adaptation methods. To delve deeper, in Figure 12 in the appendix, we also visualized the evolving interactions among learners for each optimization step by examining the attention patterns across the learners within MSFTs. In short, it reveals that learners tend to initially interact extensively with each other (high off-diagonal attention scores in the beginning) through MSFTs, followed by gradually focusing more on their individual states towards the end (high diagonal attention scores in the end). This analysis further underscores MIA's adaptability in ensuring a balanced utilization of multimodal interactions, emphasizing the necessity of such an iterative multimodal adaptation scheme.