Skill Expansion and Composition in Parameter Space

We sincerely appreciate the reviewer for the constructive comments on our work! Regarding the concerns from the reviewer oGS2, we provide the responses as follows:

W1: More explanation about LoRA in Abstract...

• Thanks for this valuable comment! We have extended the discussion about LoRA in the Abstract. Please see the Abstract for details.

W2&Q1.1: I think the paper lacks general motivations and overview on the benchmarks used...

Sorry for the confusion. Here, we summarize our motivations again. Specifically, we evaluate different axes of PSEC abilities through different benchmarks.

• First, we evaluate whether the context-aware modular framework of PSEC can effectively compose multiple skills. We test this in a multi-object composition setting (Section 4.1) using a safe offline RL benchmark, where objectives like maximizing reward, minimizing cost, and staying close to behavior policy must be jointly addressed. Unlike traditional offline RL benchmarks that focus on a single objective, this setup better assesses PSEC's compositional capabilities.
• Second, we assess whether PSEC can iteratively evolve as it incorporates more skills (Section 4.2). This is demonstrated using the DMC control suite, progressing from standing to walking and eventually running, showcasing PSEC's ability to continually learn and adapt.
• Finally, we test PSEC's applicability across diverse scenarios (Section 4.3). To do this, we evaluate its performance in handling dynamics shifts by modifying the transition dynamics of an offline RL benchmark.

Please refer to the first paragraph of each section (Section 4.1~4.3) for more details.

W3. I think one of the possible limitations of composing skills as a weighted some of LoRA actions is that some skills are not simply some of sub-skills.

• Thanks for this valuable comment! The reviewer mentions an interesting skill of throwing a ball. This skill may not be composable from others but can still act as a sub-skill for more complex tasks, such as playing basketball. PSEC can address these non-composable skills by integrating them as new LoRA modules, allowing them to be combined for more complex tasks.
• We provide more experimental results on more complex tasks in the Meta-World benchmark, covering a total of 50 skills. Please see general response and Appendix F in our revised paper for details.

Q1.2&Q1.3: What are the tasks and what do the recorded data look in the offline benchmarks? Why would they provide a characteristic examples to highlight the pros and cons of PSEC?

• For the high-level motivations, please refer to the responses to W2 & Q1 for details.
• For benchmark details, we have added more illustrations and discussions about the safe offline RL in Appendix E in our revised paper. More details about the DMC control suite, dynamics shift, and Meta-World evaluations can be found in Appendix C.2, C.3, F, respectively. Please check them for details.

We sincerely thank the reviewer for the valuable feedback. Please let us know if we have adequately addressed your concerns. If so, we would greatly appreciate it if the reviewer could consider adjusting the score.

Thanks for the replay and for increasing the score to a 6! It's happy to hear that we've successfully addressed your concerns!

We sincerely appreciate the reviewer for the constructive comments on our work! Regarding the concerns from the reviewer 1bSW, we provide the responses as follows:

W2: A potential limitation of the authors' proposed framework is its applicability only to environments where data is available...

Thanks for this valuable comment!

• We extend PSEC to a zero-shot generalization setting on the MetaWorld benchmark. Please see general response and Appendix F in our revised paper for details. Briefly, we jointly train PSEC's compositional network on 18 tasks and test PSEC on 12 unseen tasks. Results show that the compositional network can generalize well on unseen tasks, primarily due to the flexible combination and invocation of previously learned skills in the parameter space, which allows it to generalize to a certain extent to unseen scenarios.
• In addition, all experiments in our paper inherently demonstrate few-shot adaptation/transfer, since the adaptations use solely limited data, such as the transfer from standing to walking to running, use only 10 trajectories, respectively.
• To further illustrate this efficiency, we conduct a series of experiments on MetaWorld with more complex tasks. Please see general response and Appendix F in our revised paper for details.

W3: ...this may lead to confusion with existing skill learning concepts.

• Good point! In this paper, each skill corresponds to a LoRA module for each task, unlike other works where skills are defined by temporal abstractions. PSEC’s skill definition allows for more flexible skill library expansion and management, since we can directly add new LoRA modules for new tasks. However, as discussed in Appendix A (Redundant Skill Expansion), this can lead to redundant growth. To mitigate this, criteria for expansion can be implemented, such as assessing skill diversity, which is exactly mentioned by reviewer 1bSW in these relevant papers.

Q1: How does PSEC compare to the baselines in terms of computational costs and memory usage?

• In the following table, we report the training time, steps and trainable parameters of PSEC and the SOTA baseline in the safe offline RL benchmark, FISOR, for illustration.

• FISOR requires 1M training steps to train a stack of large MLPResNet blocks with 9.7M parameters. In contrast, PSEC only trains 3 small networks: 2 LoRA modules for reward maximization and cost minimization, and 1 compositional network, totaling just 0.8M parameters. This results in significantly fewer training steps and reduced training time for convergence.
• Also, we've included some of these discussions in our first submission, please see Appendix D.2 for details.

We sincerely thank the reviewer for the valuable feedback. Please let us know if we have adequately addressed your concerns, and feel free to share any additional questions or comments.

We sincerely appreciate the reviewer for the constructive comments on our work! Regarding the concerns from the reviewer 18jx, we provide the responses as follows:

W1. ...Not everyone would call this "rapid"

• Thanks for this valuable comment. We've revised our paper accordingly.

W2. the argument of parameter-level composition being superior is not well-grounded ...

• We have provided qualitative experiments in Figure 4 and discussed them in line346-line351. The discussion from line 256 to line 304 is an intuitive explanation.

W3: The t-SNE plot (Figure 4) does not directly reveal that the parameter space "shares knowledge"...

• Thanks for the valuable comment! We have added a new visualization in Figure 19 in Appendix G of our revised paper. This figure inspects whether PSEC’s final combined running skill retains behaviors from its component skills including standing and walking. Specifically, we summarize the run/walk/stand reward distributions for the trajectories rollouted by PSEC, NSEC, and ASEC. The results show that PSEC achieves high rewards across all tasks, demonstrating that its running skill retains behaviors from walking and standing, suggesting superior skill sharing compared to NSEC and ASEC.
• Additionally, there seems to be some misunderstanding for the reviewer. Figure 4(b) illustrates that different skills are clustered together chaotically, indicating suboptimal grouping due to overly overlapping skills. In contrast, Figure 4(a) preserves the distinctions between different skills, showing disjoint spaces that maintain clear separations while also revealing a shared structure, suggesting potential shared knowledge across these skills.

W4: a lot of terms that authors either didn't specify clearly ...

Thanks for these comments. We've revised them accordingly.

• versatility. This highlights PSEC’s applicability across diverse settings, including multi-objective composition, continual RL, and dynamics shift scenarios.
• generated. We have made the revisions; please refer to the updated paper for details.
• popular safe offline RL benchmark. We selected a safe offline RL benchmark to test PSEC's ability to compose skills into complex ones. This benchmark requires jointly maximizing reward, minimizing cost, and avoiding distributional shift to solve the final problem—ideal for evaluating compositionality, as improper composition can lead to suboptimal results. Specifically, we evaluated 9 autonomous driving and 8 locomotion tasks from the DSRL benchmark, where each skill has clear physical meanings, as shown in Figure 5. See Appendix E in the revised paper for more details.
• Safety. We clarified this in the caption of Table 1, following the DSRL benchmark’s definition of safety as a normalized cost below 1.
• related works. We appreciate this comment and have expanded the discussion on related works in Appendix B in our revised paper. Specifically, to the best of our knowledge, our paper is the first that systematically investigates the advantages of skill composition and expansion in parameter space over noise and action space over 4 benchmarks, covering , offering a clear guidance for future reseaches in this field.
• grammatical errors. We have carefully revised the paper to address grammatical issues and will continue to proofread it to minimize such errors.

Thank you for your responses. I'm inclined to raise my score a bit. But for the paper to be considered for acceptance, I think it's essential that there be a more thorough treatment of related work, as indicated in my review.

We really appreciate your time engaged in the review and rebuttal phase and we feel our work has been greatly improved through your constructive comments! Regarding the remaining concerns on the literature review, we thoroughly examined over 30 modularized multitask learning and continual learning methods and tried our best to summarize the differences across three key axes: how to obtain different modules, how to compose modules, where to compose modules. Please see the revised Appendix B (blud text) for details. Due to time limits, we'll continue to include more relevant works and refine our writing to provide a more concise version.

Here, we list some newly added references.

[1] Meta-neural networks that learn by learning. 1992.

[2] Design and evolution of modular neural network architectures. 1994.

[3] Manipulation task primitives for composing robot skills. 1997.

[4] A Bayesian/information theoretic model of learning to learn via multiple task sampling. 1997.

[5] Multitask learning. 1997.

[6] Modular neural network classifiers: A comparative study. 1998.

[7] Modular neural networks: A survey. 1999.

[8] Multitask learning: A knowledge-based source of inductive bias. 1993.

[9] Combining artificial neural nets. 1996.

[10] Task clustering and gating for Bayesian multitask learning. 2003.

[11] Modular deep belief networks that do not forget. 2011.

[12] Twenty years of mixture of experts. 2012.

[13] Trace norm regularised deep multi-task learning. 2016.

[14] Neural module networks. 2016.

[15] Meta-learning for fast adaptation of deep networks. 2017.

[16] Overcoming catastrophic forgetting in neural networks. 2017.

[17] Modular meta-learning in abstract graph networks for combinatorial generalization. 2018.

[18] Progress & compress: A scalable framework for continual learning. 2018.

[19] Meta-learning probabilistic inference for prediction. 2018.

[20] Experience replay for continual learning. 2019.

[21] Gradient surgery for multi-task learning. 2020.

[22] Continual deep learning by functional regularisation of memorable past. 2020.

[23] Conflict-averse gradient descent for multi-task learning. 2021.

[24] The logical options framework. 2021.

[25] In a nutshell, the human asked for this: Latent goals for following temporal specifications. 2021.

[26] An introduction to lifelong supervised learning. 2022.

[27] Combining parameter-efficient modules for task-level generalisation. 2023.

[28] Lorahub: Efficient cross-task generalization via dynamic lora composition. 2023.

[29] Solving Continual Offline Reinforcement Learning with Decision Transformer. 2024.

[30] Merging decision transformers: Weight averaging for forming multi-task policies. 2024.

Thanks for your time and valuable feedback again! Please feel free to let us know if the reviewer has further comments. It would be really appreciated if the reviewer could consider re-evaluating our paper.

We sincerely appreciate the reviewer for the constructive comments on our work! Regarding the concerns from the reviewer ac7Q, we provide the responses as follows:

Weakness: It would be also good if authors could provide their though contrasting SPEC with existing works on RL agents...

• Thanks for this valuable comment! We've added discussions on all related works [1-6] mentioned by the reviewer accordingly, along with additional relevant works, in Appendix B. Here’s a brief overview of the differences:

  • [1-2] use LoRA for multi-task learning but focus on static tuning, which significantly limits their flexibility and adaptability in scenarios requiring real-time compositional adjustments. In contrast, PSEC combines different LoRA via a context-aware modular, maximizing the expressiveness of pretrained skills to flexibly compose new skills. For example, in Figure 5, PSEC adaptively prioritizes safety over reward to prevent collisions.
  • Similar to [1-2], [4] and [6] focus on static setups such as language models, whereas PSEC targets dynamic decision-making tasks, necessitating a context-aware modular for adaptive skill composition.
  • [3] uses logical options for skill composition but requires significant human effort for skill management, limiting scalability. Additionally, [3] focuses on efficient pretraining, not on fast adaptation/continual improvement. In contrast, PSEC targets the later setups, and minimizes human effort by incorporating new skills as LoRA modules, which are then combined through auto-learned compositional networks.
  • [5] addresses a fundamentally different problem than PSEC. So, the arguments from [5] are inapplicable here. Specifically, [5] focuses on improving model pretraining for enhanced OOD generalization, emphasizing the need for shared information bottlenecks across skills for strong representations. In contrast, PSEC focuses on fast adaptation/continual improvement, where parameter-isolated LoRA modules are essential for learning new skills, especially for preventing catastrophic forgetting.

• It is difficult to ensure a fair comparison to the baselines mentioned by the reviewer for two main reasons: different settings and lack of open-source code.

  • Specifically, [1,3-6] focus on different problems (e.g., language model, static settings) compared to the one in PSEC (continuous control, adaptive composition is crucial), making them unsuitable for comparison.
  • The only comparable baseline that focuses on a similar task setting to PSEC is [2]. However, since its code is not publicly available, reproducing it for a fair comparison is difficult.

• Therefore, we compare PSEC with the latest strong baseline L2M[7], which studies in a similar setting about efficient finetuning and continual learning via multi-task offline RL pretraining. Additional experiments comparing PSEC with L2M are included in the general response and Appendix F in our revised paper.

Reference:

[7] Schmied, Thomas, et al. "Learning to Modulate pre-trained Models in RL." Advances in Neural Information Processing Systems 36 (2024).

We sincerely thank the reviewer for the valuable feedback. Please let us know if we have adequately addressed your concerns. If so, we would greatly appreciate it if the reviewer could consider adjusting the score.

• How to leverage previous knowledge to adapt to new skills?

  • [1] directly initialize from the pretrained LoRA and finetune it.
    • Limitation: It will cause catastrophic forgetting. After finetuning on new tasks, the performance on the previous tasks may be dropped.
  • [2] manually merges different skills before fine-tuning LoRA.

    • Limitations:

      1.Manual Weight Dependency: Task performance depends on manually determined weights between the previous model and the current task model. This can lead to unstable training if the merged parameters are hard to manage.

      2.Suboptimal and Unstable Skill Composition: Manually composing skills may fail to leverage shared structures across skills, potentially overwriting learned knowledge and resulting in suboptimal performance, especially when some skills are suboptimal caused by limited data.

  • PSEC offers flexibility in adapting to various settings for new tasks.
    1. Composition Stability: PSEC can avoid catastrophic forgetting by adhering a parameter-isolation manner.
    2. Composition Optimality: PSEC avoids manually composing via a context-aware modular optimized via gradient descent. This ensures efficient skill integration and preserves task-specific knowledge.

• To demonstrate this, we provide more empirical evidence by evaluating the MetaWorld benchmark (18 pretrained tasks and 12 unseen tasks):

Setting: We implemented skill modeling as described in [1], setting the skill inventory size to 18, matching the pretraining tasks. The model is pretrained on datasets from 18 Meta-World tasks, jointly optimizing the skill allocation matrix and skill inventories, with each skill represented as a LoRA module. The evaluation results for these 18 pretrained tasks and an additional set of 12 unseen tasks in the zero-shot setting are summarized in the following table.

18 pretrained tasks

12 zero-shot tasks

We also compare PSEC against the method described in [2]. The method in [2] involves training an additional model with FULL parameter for each task and manually merging all parameters beyond the LoRA components. We evaluate it on the 12 tasks in Meta-World with few-shot setting.

12 few-shot tasks

Analysis:

The results demonstrate PSEC’s superior performance over [1] and [2] across 18 tasks and 12 unseen tasks. Specifically:

• Zero-shot Setting: [1] suffers from its potentially entropic latent skills, which struggle to generalize to unseen tasks. Also, the inferior results on the 18 tasks can be attributed to parameter conflicts from training all LoRA models together, destabilizing learning.

• Few-shot Setting: PSEC outperforms [2] on the 12 unseen tasks. The additional model in [2] needs to tune far more parameters than PSEC and thus faces difficulties in the few-shot setting, leading to suboptimal performance. Furthermore, the manual merging of skills in [2] fails to leverage shared structures across skills and risks at overwriting previously learned knowledge.

These results highlight PSEC’s efficiency and robustness in utilizing prior knowledge, avoiding parameter conflicts, and maintaining strong performance in both zero-shot and few-shot settings compared to previous methods.

• Unseen Tasks Setting. To further evaluate the efficiency of PSEC on more challenging tasks, we pretrain on fewer (18) tasks and evaluate it on more (12) unseen tasks than the first setting. Firstly, we pretrain and finetune on 18 tasks to obtain 18 LoRA modules. The performance on the 18 pretrained tasks is shown in the following Table. Results show that PSEC can achieve enhanced skill learning even when the pretrained model is combined with one LoRA for each task if the skill is composed in parameter space.

• Then, we evaluate PSEC with the obtained 18 LoRA modules on the unseen tasks. For the unseen tasks, we conduct two types of experiments: few-shot setting and zero-shot setting.

Few-shot We perform few-shot learning by training the context-aware module for 1k steps using only 10% of the total data available for unseen tasks. This setup simulates scenarios with limited data on new tasks. The results, summarized in the following Table, demonstrate that PSEC achieves a high success rate on unseen tasks. This indicates that PSEC can effectively adapt to new tasks, showcasing its capability for rapid transfer learning and efficient adaptation in data-scarce environments.

Zero-shot In this setting, no data from the unseen tasks is used to train the context-aware compositional modular. Instead, this module is trained for 2k steps using datasets from 18 pre-trained tasks. It is then evaluated directly on 12 unseen tasks. Interestingly, even without access to any data from the unseen tasks, PSEC demonstrates strong performance on several tasks. Notably, PSEC substantially outperforms NSEC and ASEC on this zero-shot transfer setting, highlighting the advantages of skill compositions in parameter spaces over noise and action spaces. Overall, the results demonstrate PSEC’s ability to effectively utilize knowledge from previously learned skills to achieve strong zero-shot transfer.

Reference:

[1] Wolczyk, M., Zajkac, M., Pascanu, R., Kucin ́ski, L., and Milos ́, P.. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems (2021)

[2] Schmied, Thomas, et al. "Learning to Modulate pre-trained Models in RL." Advances in Neural Information Processing Systems 36 (2024).