Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings

We are grateful for the reviewer’s encouraging comments and thoughtful suggestions, and we address the raised points in detail below.

(Weakness 1)

Thank you for your positive comments on the soundness and empirical results of our paper. We would like to further clarify the novelty and motivation behind our method. Although our framework involves known components (e.g., the general architecture of a hypernetwork for parameter generation), it introduces key conceptual and technical innovations that set it apart from prior works. As stated in the Introduction, the key insight of our work is that task relationship can provide prior information of the task space structure in the continual learning context. Upon reviewing prior literature, we observed that most existing strategies are primarily performance-driven and heavily rely on training procedures, with limited consideration of prior inter-task information. Motivated by this gap, we proposed the use of task embeddings to guide the learning process, and designed a transferability-aware task embedding that is compatible with continual learning constraints—computed without accessing past data or models—thereby providing a principled way to inject task structure priors into the training process. Specifically, on the basis of the transferability metric H-score, we 1) proposed to use HGR divergence as a unified definition of self transferability conforming to the H-score theory framework; 2) introduced AHP normalization for transferability scale alignment across different tasks; 3) defined an optimization problem to distill the transferability information into a low-dimensional task embedding，enabling the incorporation of task-level information into continual learning strategies.

Moreover, our use of existing components is not a simple reuse or combination; rather, we carefully adapt and integrate them into our framework to serve our specific goals. For instance, compared to conventional applications of HyperNetworks in continual learning, we reformulated the HyperNet pipeline to improve its computational efficiency and enabled it to generate only LoRA parameters. These modifications substantially reduce both training and inference overhead, which is particularly valuable in modern continual learning scenarios that demand lightweight adaptation, setting our approach apart from most previous uses of HyperNetworks, which typically generate full model weights and thus incur significantly higher storage and compute costs.

As for the listed concerns, we address them below:

• Explanation for Fig. 3a Performance Variation and Impact of Introducing H-Embedding:

  In the current version, Fig. 3a presents single-seed results. The performance drop of FAA “w/o Hemb” on the ImageNet-R 10-task benchmark suggests a degree of robustness issue with vanilla HyperNet when generating LoRA parameters across different random seeds, which also appears on 5-task and 20-task benchmarks. To provide a more stable and comprehensive comparison, we additionally report multi-seed results of all three task settings in the Table, including mean and standard deviation. It can be observed that introducing our task-guided embeddings helps stabilize the performance and improves the overall average. We plan to further investigate the underlying cause of this phenomenon in the future.

Regarding the significance of performance improvement, in average our improvements compared to removing H-embedding are notable and consistent. For example, in ImageNet-R benchmark, recent state-of-the-art method (SD-LoRA) improves upon hypernet by around 1.95%, while our method achieves an additional 4.06% improvement over it and 6.09% over the original HyperNet (measured by averaged performance across 7 seeds and 3 task splits). Also, in our experimental studies, the performance gain of our framework is consistently observed across different backbones (e.g., CNN, ResNet, ViT) and benchmarks (e.g., CIFAR100, ImageNet-R, DomainNet), suggesting that our task-relation-guided prior offers meaningful benefit.

We will clarify these points in the revised paper and include modify Fig. 3a to include multi-seed results for to better illustrate robustness.

• AHP normalization:

  As mentioned in our response regarding the design of H-embedding, the introduction of AHP is primarily motivated by the need to perform transferability scale alignment across different tasks, which we believe is an intuitive and interpretable design. This design helps prevent scale mismatch with the embedding distances and ensure that the transferability signals remain comparable across tasks when used as guidance.

  In essence, AHP serves as a normalization post-processing step for the transferability values, rather than introducing new information. This may partly explain its relatively modest quantitative impact. Nevertheless, incorporating AHP still leads to an empirical performance gain of approximately 4% (averaged across 3 task splits). While this improvement is not decisive, it demonstrates that AHP provides a non-trivial contribution to the final performance and justifies its inclusion in the framework.

(Weakness 2)

Regarding the experimental setup, we would like to clarify that all our experimental benchmarks—both the datasets and their specific splits—are directly adopted from existing works that are well-established and repeatedly used in the recent literature (e.g. [Liang and Li, 2024], [Wu et al., 2025]). The architectures we employ, ViT-B/16 and ResNet-32, are also widely used and considered representative choices in prior studies. Therefore, we do not propose any new test beds in this work; this design choice was intentional to ensure that comparisons with previous methods are fair and objective.

(Weakness 3)

The baselines for full-parameter ResNet experiments (including CIFAR-100 and ImageNet-R benchmarks) were reproduced by us, while the baselines for ViT-based parameter-efficient fine-tuning (PEFT) were cited from existing literature. In particular, we mainly cited results from the SD-LoRA [Wu et al., 2025] paper (including L2P, DualPrompt, CODA-Prompt, HiDe-Prompt, InfLoRA, and SD-LoRA), and the results of HiDe-Prompt on CIFAR-100 benchmark was taken from its original paper.

This choice was made because the full-parameter methods span a relatively long time range, and many of them do not share a unified benchmark setting or backbone architecture. To ensure fairness and consistency, we reproduced all these baselines under our evaluation framework. In contrast, most of the ViT-PEFT methods are recent works with more consistent experimental setups, making it more reasonable to cite their results directly.

(Weakness 4)

Despite its seemingly complexity, the hypernet framework does not incur significant time costs or memory usage during inference, as the number of parameters in the hypernetwork has been strictly controlled. In the original hypernet framework, the number of hypernet parameters is roughly comparable (or smaller) to that of the main network. Therefore, if processing one sample at a time, a single prediction involves one forward pass through the hypernet and one through the main net, theoretically leading to an inference time approximately twice that of a regular model. However, in practice, we only need to perform one forward pass through the hypernet per task, which means that the amortized time cost per sample becomes negligible when the dataset is large. The situation is even more favorable in the ViT-LoRA setting, where the hypernet is only responsible for generating LoRA parameters. Since these modules are much smaller in size than the full ViT model, the forward computation of the hypernet contributes only marginally to the overall inference time. We conducted runtime tests for both the ResNet and ViT backbones using full testing set, and the results support our analysis.

We believe that the comparison of inference time and parameter size between our method and the plain backbone already demonstrates the efficiency of our framework at inference time. Due to time constraints in the rebuttal phase, we were unable to conduct a comprehensive comparison with all baseline methods, but we hope the above analysis and measurements sufficiently support the claim that our method introduces minimal inference overhead.

(Weakness 5)

For similar reasons as stated in analysing the inference efficiency of our framework, the training computational costs of training hypernet is also relatively minor. We list the results of 1-epoch runtime tests of ViT-LoRA backbone on ImageNet-R 10 task benchmark below:

As shown, the 1-epoch training time of our method is slightly higher than that of plain LoRA, but the difference is minor and acceptable. Given their similar parameter sizes, the memory usage is also within a reasonable range.

I sincerely thank the authors for their detailed and thoughtful rebuttal. After carefully reading the responses, as well as the comments from the other reviewers, I believe the authors have addressed the majority of the concerns raised during the initial round of review.

That said, I would like to raise a few remaining points that I believe, if addressed in the camera-ready version, would further improve the clarity and completeness of the work:

1. The use of H-score-based embeddings is motivated. However, for the benefit of readers, I would encourage the authors to either:

• Include a brief ablation study substituting H-score with an existing embedding method (eg. Task2Vec [1] or LEEP [2], see Reviewer JmhA comments), or
• If such experiments are not feasible at this stage, add a concise discussion explaining why these alternatives may not be suitable for your setting.

2. While the authors have shown that the training and inference time overhead is relatively minor, it would be valuable to also report peak GPU memory consumption during both training and inference, so to have a more complete view of the method's requirements.

If the authors are able to incorporate these remaining points into the final version, I would be inclined to raise my score.

References:

[1] Achille, Alessandro, et al. "Task2vec: Task embedding for meta-learning." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[2] Nguyen, Cuong, et al. "Leep: A new measure to evaluate transferability of learned representations." International Conference on Machine Learning. PMLR, 2020.

We sincerely thank the reviewer for the thoughtful follow-up and the constructive suggestions. We are glad that the core concerns have been addressed and appreciate the opportunity to further improve the clarity of our work. We address each of the two remaining points below:

(1) On the use of H-score versus other task embedding methods:

Regarding the suggestion to consider alternative task embedding strategies, we appreciate this insightful point. We would first like to clarify the rationale behind our proposed H-embedding and the use of H-score within our framework. While technically task embeddings can be estimated using various methods, the alternatives are generally less suitable for our target setting, as they either require access to source data for each task—assumptions that are infeasible under rehearsal-free CL—or do not explicitly model task-to-task relationships, thereby offering limited benefit in facilitating bidirectional transfer across CL tasks.

To address this, we propose a new formulation where task embeddings are computed online by explicitly optimizing them to approximate a transferability metric across tasks. In this design, we adopt H-score as the target transferability metric primarily due to its source-data-free nature, which allows us to maintain compliance with CL constraints. Additionally, H-score is theoretically grounded, computationally efficient, and empirically shown to correlate well with actual transfer performance. These characteristics make it a particularly suitable and representative choice for our embedding algorithm. Yet, our framework is not inherently tied to H-score, and there admittedly exist other source-free embeddings or transferability metrics (e.g. task2vec, LEEP, LogME) that could also, in principle, be used within it.

As both an ablation and a further assessment of our framework's flexibility, we conducted preliminary experiments on ImageNet-R 10 task benchmark by:

• Replacing H-score with other source-free transferability metrics (LEEP and LogME) while keeping the embedding algorithm and overall pipeline unchanged;
• Replacing the entire H-embedding formulation with Task2Vec.

The results are shown below:

As shown, replacing the transferability metric alone leads to only minor drop in performance. This supports our view that while H-score is a strong instantiation, the core contribution of our work lies in the transferability-guided embedding formulation and its integration into a hypernetwork-based parameter generation pipeline—a general framework that remains valid even when alternative metrics are employed.

However, replacing the entire embedding formulation with task2vec leads to a more noticeable performance degradation. We believe this is because Task2Vec computes task embeddings independently without explicitly modeling task-to-task relationships, which limits its capacity to guide transfer dynamics in CL. Additionally, Task2Vec is highly sensitive to the choice and training of the probe network, which introduces extra variance in effectiveness and incurs additional memory costs—challenges not present in our H-embedding approach.

Due to the limited time available during the rebuttal phase, we focused on a representative subset of alternatives rather than attempting an exhaustive evaluation. We will incorporate the above results into the final version along with a more comprehensive discussion. Further exploration of other embedding designs also remains an important direction for our future research.

(2) On memory usage:

We agree that this is an important consideration. In our internal profiling, we observed that the peak GPU memory usage of our method is slightly lower than that of the vanilla ViT backbone during inference (ours: 5808 MiB vs. ViT-LoRA original: 5832 MiB), but approximately twice as high during training (ours: 24,970 MiB vs. ViT-LoRA original: 13,022 MiB). These values are rough measurements based on a fixed batch size of 128 on ImageNet-R using NVIDIA A800 GPUs. Due to time constraints, we have not yet profiled other datasets or configurations. We note that actual GPU memory consumption can vary depending on batch size, PyTorch's memory allocation behavior, and implementation details. The higher training-time usage in our method is likely caused by additional memory demands in loss computation and backpropagation, which we have not specifically optimized. Since memory was not a primary bottleneck in our experiments, we did not focus on its minimization. Nevertheless, we recognize the value of reducing memory overhead, and are open to explorations of more efficient engineering solutions in the future.

I thank the authors for their comprehensive and timely response. All of my primary concerns have now been satisfactorily addressed. Accordingly, I have increased my score.

We sincerely appreciate the reviewer’s positive evaluation and the insightful feedback provided on our work. Below, we address each of the points in detail.

(Weakness 1 & Question 2)

We agree that scaling full-parameter generation via hypernetworks to very large models can raise efficiency concerns. To address this, one of the key contributions of our work is to combine current hypernet frameworks with parameter-efficient fine-tuning (PEFT) techniques. Specifically, we employ the hypernet to generate LoRA parameters instead of full weights in our current design, with potential for future extensions to other module types. This design enables our method to be flexibly integrated into a wide range of large-scale models as a plug-in module, allowing us to leverage the capabilities of powerful pretrained backbones while keeping the hypernetwork lightweight and efficient. As one of our main comparison studies, the HyperNet-LoRA framework has demonstrated strong and stable performance across our experiments, confirming the effectiveness of this design.

(Weakness 2)

The learned embedding indeed plays a critical role in hypernetwork-based frameworks, which is also one of the reason we propose to guide its learning through a transferability-aware objective. Yet, in our framework, the performance is relatively insensitive to the embedding dimensionality as a hyperparameter. To support this, we conducted experiments with embedding dimensions set to 8, 16, 24, and 32 on ImageNet-R 10 task ViT setting. As shown in the table below (`*' denotes the setting in experimental results), although FAA exhibits slight fluctuations across settings, the overall performance remains stable. This suggests that our method is relatively robust to changes in embedding size, and does not rely on fine-tuning this hyperparameter to achieve good results.

We appreciate the reviewer’s insightful comment regarding the characteristics of the embedding. We agree that this aspect is indeed worth further investigation, as it could provide valuable insights into the behavior of transferability-aware hypernetworks. We hope to conduct a more comprehensive study in this direction in future work.

(Question 1)

We thank the reviewer for the valuable feedback and agree that the definition in Eq. (2) can be clarified further. In future revisions, we will provide a more intuitive explanation of the formula to enhance its readability. Although the two instances of "cov" might suggest different interpretations, they are consistent from a computational standpoint — both denote the covariance computed over the first dimension (i.e., across the observations) of the input data. We will explicitly clarify this point in the revised manuscript.

(Question 3)

We appreciate the reviewer’s thoughtful observation. Firstly, we would like to clarify that compared to typical deep learning methods, our framework does not introduce significantly more “moving parts”. The two hyperparameters $\beta_e$ and $\beta_c$ serve as standard loss balancing factors, which are commonly adopted in many existing works; the encoder and decoder modules are designed as very shallow linear networks, jointly trained with the main HyperNet, making the overall architecture relatively straightforward and within the scope of typical deep learning designs.

To further illustrate this, we provide a parameter count comparison showing that the total parameter size of our HyperNet framework is comparable to that of conventional model architectures. The connections involved are also quite simple, and the training is end-to-end. So far, we have not observed any indications of local minima or poor local optima issues during experiments. Regarding the sensitivity to the hyperparameters $\beta_e$ and $\beta_c$, our empirical results suggest that the framework is relatively robust to their variations. Below we summarize the performance under different settings of these hyperparameters on ImageNet-R 10 task ViT setting (`*' denotes the setting in experimental results).

We sincerely thank you for your helpful comments and positive evaluation of our paper. We will continue to refine our work based on your valuable suggestions.

We sincerely thank the reviewer for the thoughtful and detailed feedback. Below, we respond point-by-point.

(Weakness 1 & Q1)

For the concerns about HyperNet [von Oswald et al., 2020] comparison and the performance fluctuations shown in Fig. 3a, we address both problems below.

• Performance Significance Compared to HyperNet: Despite its limitations, we acknowledge that the original HyperNet is a relatively mature CL strategy, and surpassing it by a large margin is naturally more challenging than typical ablation settings. However, we believe our improvements over HyperNet as a baseline are still notable and consistent. For example in ImageNet-R benchmark, recent SOTA method (SD-LoRA) improves upon hypernet by 1.95%, while our method achieves a 4.06% improvement over SD-LoRA and 6.09% over the original HyperNet (measured by averaging across 7 seeds and 3 task splits). Also, in our experimental studies, the performance gain of our framework is consistently observed across different backbones (e.g., CNN, ResNet, ViT) and benchmarks (e.g., CIFAR100, ImageNet-R, DomainNet), suggesting that our framework offers meaningful benefit beyond HyperNet.

• Explanation for Fig. 3a Performance: Currently, Fig. 3a presents single-seed results. The performance drop of “w/o Hemb” on the ImageNet-R 10-task benchmark suggests a degree of robustness issue with vanilla HyperNet when generating LoRA parameters across different random seeds, which also appears on 5- and 20-task benchmarks. To provide a more stable and comprehensive comparison, we additionally report multi-seed (7 in total) results of all three task settings in the Table, including mean and standard deviation. It can be observed that introducing our task-guided embeddings helps stabilize the performance and improves the overall average. We plan to further investigate the underlying cause of this phenomenon in future.

• Novelty and Methodological Contributions: While our framework adopts a HyperNet-like architecture for parameter generation, it substantially differs from the previous work in both motivation and implementation. The central contribution of our work lies in the introduction of task relation priors, which guide the dynamic adaptation of model parameters across tasks. In contrast to the vanilla hypernet, where task embeddings are learned implicitly through task-specific data, we explicitly model and compute inter-task affinities to guide the learning of task embeddings and hypernet, resulting in better generalization and stability across continual tasks.

  Moreover, as part of our contribution, we reformulated the implementation of HyperNet pipeline, improving its computational efficiency and enabling it to generate only LoRA parameters. These adjustments significantly reduce both training and inference overhead, and are especially beneficial in modern CL scenarios where lightweight adaptation is critical. It also distinguishes our work from most prior uses of HyperNet, which typically generate full model parameters or require heavier storage and compute budgets.

We will clarify these points in the revised paper with Fig. 3a including multi-seed results.

(Weakness 2)

2.1

Eq.(11) is not intended to align the additional output with $\hat{𝑒}^𝑗$, but rather to enforce alignment between $\hat{𝑒}^𝑗$ and the transformed version of $e^j$ through the encoder-decoder mapping. In essence, this alignment ensures that the guiding task embedding $\hat{𝑒}^𝑗$ can be well approximated by a learned transformation of $e^j$, or intuitively from an information transmission perspective, $e^j$ should retain sufficient information to recover the H-embedding $\hat{𝑒}^𝑗$.

The encoder-decoder structure can be equivalently viewed as an autoencoder, of which the hidden layer is coupled with the intermediate layer of the hyperNet, facilitating joint optimization and promoting a consistent representation space between embeddings and hyperNet outputs. This type of alignment using autoencoders has been widely adopted in previous literature(e.g., [Schonfeld et al., 2019]). It does not require reconstruction of the original $e^j$; rather, the core idea is to provide a bridge between the prior calculated H-embedding and the embedding being learned. We described it as an encoder-decoder module to maintain conceptual clarity and avoid confusion with the core HyperNet mechanism.

2.2

We agree that the more common formulation is: $aff = exp(-\gamma \times dist)$, yet we adopts the reverse mapping: $dist = exp(-\gamma \times aff)$. Nevertheless, this reverse form is not incorrect, and has been used in prior works too — notably in Taskonomy [Zamir et al., 2018]. Intuitively, both affinity and distance are relational measures where the relative scale matters more than the absolute value. Such a transformation is to ensure a smooth inverse relationship, and our formulation also satisfies this requirement.

The inclusion of the scaling factor $\gamma$ further enhances this effect: it adjusts the steepness of the exponential curve, ensuring the learned embedding distance emphasizes relative differences without being dominated by absolute magnitude. This is especially important in our framework where the scale sensitivity of H-score transferability across different target tasks can affect the optimization stability of H-embedding computation.

2.3

We acknowledge that transferability is inherently asymmetric, and does not naturally match the symmetry property of Euclidean distance. However, in our framework, this alignment remains a reasonable and effective approximation, due to the following:

• Asymmetric guidance is sufficient: In CL, we are primarily concerned with how previous tasks affect the current task. Thus, we only require one-way transferability and inter-task relation, and symmetric alignment is not strictly necessary.

• Constraint of CL: Since our framework operates in a strict rehearsal-free CL setting, we cannot revisit or recompute full two-way transferability after a task is finished. The one-way transferability from known tasks is the only feasible choice.

Overall, we believe this approximation still provides meaningful structure for guiding embedding space learning and has been validated by empirical improvements.

(Weakness3 & Q3)

In this work, we mainly focus on the increasingly popular rehearsal-free CL setting, where access to past data is prohibited. In contrast, [Ermis et al., 2022] relies on storing both past models and selected samples, which violates our problem setting. Therefore, this work is not currently included in our main comparisons. Nevertheless, we still thank the reviewer for bringing it up and the underlying ideas are indeed related to ours. We'll include a discussion in the revised version.

Despite the setting mismatch, we conducted an additional experiment. As the code for the paper is not publicly available and the full reproduction would require considerable effort, we instead implemented our method on the benchmark (CIFAR-100 20 tasks + ViT) used in Fig 3 of the original paper, with their method reporting a FAA of approximately 0.78 and our method achieving 0.907.

Since the original paper does not provide detailed numerical results, we estimated their performance by reading values from the figure. Also, due to differences in dataset configurations, the current comparison may not be entirely fair. We will conduct a more rigorous and fair comparison once we have sufficient time and resources in the future.

This also partially addresses Q3. Our choice of metric was driven by practical considerations. In rehearsal-free CL, it is infeasible to store large amounts of past data or model parameters between tasks. The H-score metric stands out because it requires no source data, and the only information it needs—source model parameters—can be derived using the HyperNet from the task embedding. Hence, our framework strictly adheres to the CL constraints, while still enabling fast and theoretically grounded estimation of task relationship. Additionally, H-score is one of the most currently recognized transferability metrics in the field, known for its solid theoretical backing and strong empirical correlation with true transfer performance. We believe these make it a well-justified choice.

We acknowledge that our current study does not cover the full spectrum of possible solutions. However, the presented method offers a feasible and effective approach to enhancing CL. Moving forward, we plan to explore alternative designs and conduct a more comprehensive analysis, in terms of both accuracy&efficiency.

(Weakness 4)

...

(2) Regarding W2.3 (symmetry and AHP motivation):

Thank you for raising this point, and we apologize for the misunderstanding in our previous response. Our earlier reply focused on clarifying that aligning the symmetric Euclidean distance to the asymmetric transferability is generally acceptable. Yet, this does not mean that directly aligning them during optimization does not introduce practical issues.

In particular, H-score is a target-centric transferability metric, and its absolute value is largely influenced by the specific target task. This variation misaligns with the symmetry assumption of Euclidean distance: the "score distance" from task A to task B may differ significantly from that from task B to A due to differences in target task property. To mitigate this, we introduced AHP normalization to bring transferability values across tasks into a more consistent range. This normalization helps reduce the scale mismatch and promotes better structural alignment between transferability and Euclidean distance, thus facilitating the optimization of H-embeddings.

We acknowledge that our original explanation (Lines 159–162) may have been unclear and will revise it in the final version to better reflect this motivation.

(3) Regarding W2.2 (affinity-to-distance conversion):

Thank you for the further discussion on the affinity-to-distance conversion. We address the points as follows:

• We are sorry for the lack of clarity in our earlier response. Following the question raised in W2, our previous reply focused on whether reversing the affinity–distance direction was reasonable, without carefully distinguishing the placement of $\gamma$. In our implementation, we actually use $dist = \gamma \cdot \exp(-aff)$. This choice provides a more intuitive scaling in our setting, where the converted distance is then used for alignment. Also, when $\gamma$ is around 1 (as is the case for most of our optimization results), the two forms behave similarly in practice.

• It appears in caption of Fig. 3 of Taskonomy [Zamir et al., 2018].

• Thank you for pointing this out. We apologize for the inadequate statement and will revise it for better precision, along with adding necessary discussions in the final version.

• While we agree that $dist =\exp(-aff)$ can reduce sensitivity to high affinity values, in our setting it is still a reasonable choice, in that: i) H-score and its AHP-normalized version can be sometimes negative, and the exponential ensures positive distances; ii) extreme affinity values may arise from numerical artifacts, so lower sensitivity in this case can be beneficial; iii) within the normal range, the narrowing effect is minor since the derivative of $\exp(-x)$ is -1 near zero. Nevertheless, we acknowledge that there is still room to further investigate the properties of this conversion function, and we thank the reviewer for bringing up this valuable point.

Overall, we recognize that there is room for refinement in the current presentation of our affinity-to-distance conversion, and we will do our best to address these points in the final version to improve rigor. At the same time, we would also like to note that this technical aspect is not central to our framework and does not affect the validity of our main contributions.

(4) Regarding W2.1 (missing citation):

The full reference is Schonfeld, Edgar, et al. "Generalized zero-and few-shot learning via aligned variational autoencoders." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019., where they included a section of related work on Cross-Reconstruction. We will ensure that it is properly included in the final bibliography.

We would like to thank the reviewer again for the valuable feedback, which has helped us strengthen the clarity and completeness of the paper. Please let us know if further clarification is needed.