5. Complexity and Computational Costs

.

The proposed method is complex and may bring additional implementation and computational costs.[...] Could you provide details on the run-time and computational efficiency of the proposed algorithm? Given the added complexity from subspace extension, anchor sampling, and hierarchical imitation learning, how does HILOW’s efficiency compare to that of the baselines ?

We understand the concerns regarding the computational complexity of our method. To clarify, HILOW does not introduce significant additional complexity compared to most baseline methods such as FT1, FTN, SC1, SCN, EWC, and L2 (Figure 15). In HILOW, at each step, we train new anchors similarly to how these previous methods handle new tasks by updating or maintaining their policies. The overhead added is mostly due to the evaluation of the sample anchor weights in the considered subspace of policies.

The primary exceptions are CSP and PNN. The first introduces the learning of a value function which significantly slows down the computation. The second one is an ever growing method, which introduces learned connectors using previous features maps, and thus requires the previous parameters to be used for inference while training. For environments with smaller input size the overhead is not that significant, however for the Godot ones, due to the high dimensionality, the overhead is significant.

4. Explanation of Anchor Weight Sampling from Dirichlet Distribution

.

The choice to sample anchor weights from a Dirichlet distribution lacks adequate explanation. While Dirichlet distributions are commonly used for weight sampling, relying solely on this approach may not be sufficient for finding the optimal weight combination in complex tasks. A stronger justification or alternative exploration method could improve this aspect.

We appreciate your feedback regarding the computational complexity of our HILOW framework. To address your concerns, we conducted additional experiments comparing HILOW's run-time and memory efficiency against baseline methods across various task streams. The detailed results and analyses have been added to the appendix [E.1. and E.2.]

Our choice to use a Dirichlet distribution is deliberate and serves a specific purpose in our framework. Specifically, we aim to uniformly sample weights within a given simplex, ensuring that the anchor weights are non-negative and sum to one. To achieve this, we utilize a Dirichlet distribution with equal parameters for each dimension [15], which facilitates uniform exploration across the simplex. In practice, we simulate such sampling methods using a stick-breaking process for faster sampling in Python, which further accelerates the evaluation [16]. The Figure 14. shows how fast it is to sample alphas, and how well it covers a considered anchor weights simplex (in the case where we have 3 anchors represented as the summits of the triangle).

Uniform sampling in the simplex is crucial for covering a diverse range of weight combinations, which enhances the robustness of our policy subspaces. Experimentally, we have found that by sampling a high number of anchor weights, we effectively cover the simplex, allowing our offline evaluation process to rank and select the most effective combinations. This approach ensures comprehensive exploration without biasing towards any particular region of the simplex.

While alternative methods such as gradient descent over the simplex could be considered, they introduce additional computational costs and the risk of converging to local minima. Our Dirichlet-based sampling method strikes a balance by providing extensive coverage of the weight space with manageable computational overhead, making it well-suited for our offline evaluation framework.

[15] Dirichlet and related distributions: Theory, methods and applications, K. W. Ng et al.

[16] A simple proof of the stick-breaking construction of the dirichlet process, J. Paisley et al.

3. Choice of the baseline

.

The baselines used in the experiments are designed for continual reinforcement learning. Is it fair to compare these with the proposed method in goal-conditioned offline continual reinforcement learning? Could you explain why these baselines were chosen and how well they address the challenges specific to goal-conditioned offline continual reinforcement learning tasks?

Thank you for raising the point regarding our choice of baselines. Currently, there are no established methods explicitly designed for goal-conditioned offline continual reinforcement learning, nor are there standardized benchmarks for evaluating such methods. To address this gap, we adapted well-known continual learning strategies from both supervised and reinforcement learning domains — such as Elastic Weight Consolidation (EWC), L2 regularization, Fine-Tuning (FTN), and Progressive Neural Networks (PNN) — as baselines in our experiments. These methods are foundational adaptive strategies widely used to mitigate catastrophic forgetting and facilitate knowledge retention across sequential tasks.

We selected these baselines because they have been extensively employed in related continual reinforcement learning research, including studies on Continual Subspace of Policies (CSP), which our work extends to multiple subspaces applied to the goal-conditioned setting. By applying Hierarchical Imitation Learning as the backbone for both high-level and low-level policies across all strategies, we ensured a consistent and fair comparison. Although these baselines were not originally designed for goal-conditioned tasks, their inclusion underscores the necessity for specialized methods like ours and demonstrates the effectiveness of our framework in advancing the field.

2. Manuscript Cohesion and Logical Flow

.

Each section of the paper reads as a list of points, with minimal cohesion or natural progression between parts. For example, there is no clear logical connection between sections 4.3 and 4.4. It is unclear why subspace exploration is needed immediately after subspace extension. This disjointed organization makes the overall flow of the paper difficult to follow and requires significant revision to improve coherence.

We understand the importance of a cohesive narrative and agree that the current structure may hinder readability. We have provided a thorough explanation in our general response and kindly refer you to that section for more details.

By refining the flow of the manuscript, we aim to create a unified narrative that clearly illustrates how each component of our approach contributes to addressing the challenges of continual offline goal-conditioned reinforcement learning.

We sincerely thank you for your thorough and constructive feedback on our manuscript. Your insights have been relevant to identify areas where we can enhance the clarity and impact of our work. We address your concerns below the revisions we propose to make for the camera-ready version.

.

1. Highlighting the Focus on Goal-Conditioned Navigation Tasks

.

This paper focuses on continual offline reinforcement learning with an emphasis on goal-conditioned navigation tasks, but this specific problem is not adequately highlighted throughout. The title, as well as the related work and preliminaries sections, do not mention goal-conditioned learning explicitly. Additionally, the paper does not clearly define the continual offline reinforcement learning problem with goal-conditioned navigation tasks or explain its importance, which weakens the motivation behind this work.

Thank you for pointing out the need to emphasize our focus on goal-conditioned navigation tasks. We acknowledge the need to more prominently highlight our focus on offline goal-conditioned navigation tasks. While goal-conditioned learning is discussed in the introduction [Lines 036 - 073 - 074], we agree that it should also be emphasized in the related work section. Thus, we propose to enhance the related section to include a dedicated subsection of goal-conditioned reinforcement learning, detailing its significance and how it integrates with continual learning. Finally we are considering adding this precision in the title as : Hierarchical Subspace of Policies for Goal-Conditioned Continual Offline Reinforcement Learning.

Thank you for your response. I’ve gone through the new version, and I find it much clearer than the previous one. Most of the issues I mentioned have been addressed. I’m willing to raise the score to 6.

1. Comparison with Baselines

.

FTN and SCN seems to be a strong baseline as seen in Figure 3 with similar performance to memory size tradeoff. Additionally, approaches such as Polyoptron, Multitask prompt tuning and similar peft approaches have been found to help with multi-task adaptation which may additionally help with memory efficiency and would be a good standard of comparison alongside these baselines.

We appreciate your suggestion to include comparisons with additional baselines. However, methods such as Polyoptron and Multitask Prompt Tuning are effective in NLP and supervised learning, focusing on prompt-based and parameter-efficient fine-tuning techniques that leverage labeled data and language model architectures. Adapting these approaches to the domain of continual offline reinforcement learning would require significant modifications to address the sequential, goal-conditioned nature of our tasks and the absence of interactive data collection.

3. Related Work Revisions

.

Irrelevant Related Work : The related work section includes extensive discussions on topics like transfer learning, multitask learning, meta-learning, and online continual reinforcement learning, which are not directly pertinent to this paper. Conversely, the core techniques employed, CSP and LoRA, are not adequately covered, leaving a gap in the context for readers.

We believe precisely positioning our work among related learning paradigms is both important and valuable to grasp the limits of our work. Nevertheless, we acknowledge that the related work is missing discussions and positioning over most related work. In the revised manuscript, we propose to refocus the related work section for this purpose, and invite you to read the provided general answer.

Specifically, to address your concerns, we have thoroughly revised the Related Work section by consolidating discussions on Transfer Learning, Multitask Learning, and Meta-Learning into a single, concise paragraph, thereby maintaining a sharp focus on Continual Reinforcement Learning (CRL).

We have also expanded our coverage of core methodologies such as Continual Subspace of Policies (CSP) and Low-Rank Adaptation (LoRA), providing detailed explanations of their relevance and limitations within the CRL context. Additionally, we have clearly differentiated between Online and Offline CRL, emphasizing the lack of standardized benchmarks for offline, goal-conditioned CRL, which highlights the novelty and significance of our framework.

These revisions ensure that the Related Work section is both relevant and comprehensive, effectively positioning our contributions within the existing literature.

2. Improvement over Baselines

.

Limited Improvement : The proposed algorithm, HILOW, does not demonstrate substantial improvement over basic strategies, and the limited improvement comes predominantly from existing techniques. As shown in Figure 3, the algorithm's performance is similar to that of expanding naïve strategy (SCN) and expanding fine tuning strategy (FTN), with the primary advantage being reduced memory consumption. However, the ablation study indicates that most of the memory savings (approximately 75%) compared to CSP-O are due to LoRA, not the hierarchical subspace approach.

Questions : Could you provide more detailed analysis of experiments that isolate the impact of the hierarchical subspaces, demonstrating their specific contribution to the overall performance and memory efficiency ?

We appreciate your request for a more detailed analysis. To isolate the impact of the hierarchical subspace approach, we conducted additional experiments focusing on task streams with logical progressions, where tasks are logically related. This allowed us to demonstrate how the hierarchical subspace approach enhances adaptation and efficiency. In the following, we will refer you to the corresponding tables and figures added to the manuscript, for greater reading comfort.

.

1. Experiments on Task Streams with Structured Changes :

We focused on task streams where tasks are logically related, allowing us to demonstrate how the hierarchical subspace approach enhances adaptation and efficiency. We compared our method against baselines including Fine-Tuning (FTN) and Continual Subspace of Policies (CSPO), as well as an ablated version of our method, HSPO (HILOW without LoRA), to isolate the effect of hierarchical subspaces.

Dynamic Changes Stream (AntMaze)

In the AntMaze dynamic change stream, the agent encounters tasks with varying action dynamics while navigating the same maze topology, including umaze-normal, umaze-permute_actions, umaze-inverse_actions, and umaze-permute_actions. This setup tests the agent's ability to adapt to changes in action dynamics within a consistent environment.

As shown in Table 11. and Figure 11., HSPO achieves similar performance to CSPO while improving memory efficiency. Our hierarchical subspace approach effectively adapts to task progressions and environmental variations without redundant replication of high-level policies, unlike CSPO. This ensures efficient adaptation to new tasks while maintaining performance on previously learned ones.

The memory savings may seem modest due to the intentionally small high-level networks (selected to accelerate training). However, the hierarchical structure functions as intended, effectively scaling as we wanted and expected.

Topological Changes Stream (Godot)

This stream evaluates the agent's ability to adapt to changes in maze layouts, requiring the high-level policy to adjust its strategic planning. The tasks consist of four progressively complex configurations: maze_1-high, maze_2-high, maze_3-high, and maze_4-high.

As shown in Table 12., HSPO achieves better memory efficiency compared to CSPO, with the low-level policy being reused across tasks when they are sufficiently similar. While the performance differences between the methods are less pronounced, this stream demonstrates how HSPO's hierarchical structure effectively reduces redundancy and optimizes memory usage.

Now the relative memory savings appear greater due to the bigger size of the low-level networks (chosen for accurate movements).

.

2. Comparison of CSPO and HSPO with and without LoRA :

Both CSPO-LoRA and HILOW achieve significant memory savings through Low-Rank Adaptation (LoRA), as shown numerically in Figure 13. Our hierarchical subspace approaches (HSPO and HILOW) independently enhance performance and further improve memory efficiency by separating high-level and low-level policies, thereby reducing redundancy.

The integration of LoRA complements our framework, effectively adapting to dynamic and topological changes in the environment.

Future work could explore the interpretability of policy adaptations and the relationship between LoRA’s rank and environmental changes, with potential applications in industrial settings and unsupervised hyperparameter tuning.

We thank reviewer R#k3LF for taking the time to provide thorough and insightful feedback on our manuscript. We appreciate your recognition of the clarity of our presentation and your constructive comments regarding the novelty and contributions of our work. Below, we address each of your points in detail.

.

1. Contribution and Novelty

.

Limited Contribution : Most techniques proposed in the paper are already applied in related domains. The continual subspace of policies (CSP) is proposed as a continual reinforcement learning method, adapting it to continual imitation learning is straightforward. The low-rank adaptation (LoRA) is also well-established in continual learning, and its application here does not present any distinct innovation. The only potentially novel aspect is the hierarchical subspaces of policies.

We understand your concern regarding the novelty of our contributions. Our work is the first to consider simultaneously learning multiple subspaces of policies in an offline goal-conditioned reinforcement learning context. While CSP and LoRA are established methods, our hierarchical subspace approach introduces a novel structure that enables targeted adaptation without redundant policy replication.

Moreover, subspace-based methods for policy learning are a relatively recent development with great potential for further exploration and innovation [1,2,3,4]. Our work not only advances the state of the art by considering subspaces in parallel, but it also seeks to contribute to the foundational understanding of how subspaces can be effectively utilized in reinforcement learning. By sharing our work, we hope to spark new discussions and inspire the others to delve deeper into the possibilities of subspace representations, which hold significant promise for addressing a wide range of challenges.

It is also important to note that there are currently no established methods explicitly designed for goal-conditioned offline continual reinforcement learning, nor are there standardized benchmarks in this specific area. While continual reinforcement learning is a rapidly growing field, focusing on memory efficiency and handling complex tasks, the integration with goal-conditioned objectives, especially in an offline context, remains largely unexplored whereas it becomes a very active research area [5,6,7]. Our work also positions several baseline strategies with respect to that problematic.

To go further, we kindly refer you to the general answer we provided to all reviewers.

[1] WARP: On the Benefits of Weight Averaged Rewarded Policies, A. Ramé et al.

[2] WARM: On the Benefits of Weight Averaged Reward Models, A. Ramé et al.

[3] Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities, E. Yang et al.

[4] Subspace-Configurable Networks, D. Wang et al.

[5] Goal-Conditioned Reinforcement Learning: Problems and Solutions, M. Liu et al.

[6] OGBench: Benchmarking Offline Goal-Conditioned RL, S. Park et al.

[7] HIQL: Offline Goal-Conditioned RL with Latent States as Actions, S. Park et al.

2. Evaluation on More Diverse Environments

.

Also i think the newly introduced environments such as the Godot Maze and Godot Agent are either too close to the already existing ones or the differences are not explained well enough. The Godot Agent comes across as a simpler version of the Ant or Point agent, while the Godot Maze and MuJoCo Maze in figure 2 seems to be identical but with different colors.

We appreciate your feedback regarding the clarity and distinctiveness of our proposed environments. In complement to the general answer provide above, we wanted to present a few elements to better highlight the relevance our Godot environments :

Godot Maze :

• Raycasts : Godot Maze utilizes raycasting to generate depth maps, providing richer sensory inputs compared to the standard observations in MuJoCo environments. This enables more complex decision-making and navigation strategies.
• Configurability : The Godot Maze allows for various configurations and modifications to the maze structure, facilitating continual learning by presenting new challenges without altering the core environment. Environments can be reconfigured to present new navigational challenges while retaining core objectives, facilitating the assessment of an agent’s ability to adapt without forgetting previous tasks.

Godot Agent :

• Mobility : In contrast to agents like Ant or Point, the Godot Agent is capable of jumping and navigating over objects, adding a layer of complexity to the navigation tasks.
• Customization : The Godot Agent is specifically designed for navigation tasks within maze-like environments, enabling focused evaluations of hierarchical subspace adaptations.

While Figure 2 provides a 2D overview, which may lead to think MuJoCo and Godot mazes are similar, Figure 6 in the annex offers a 3D representation of the Godot Maze, clearly illustrating the dynamic obstacles and enhanced agent capabilities that differentiate it from the MuJoCo mazes.

By emphasizing these distinctive features, we aim to clarify the unique contributions of our Godot environments and their suitability for evaluating our framework in continual offline goal-conditioned reinforcement learning scenarios.

1. Baseline Comparisons

.

The greatest weaknesses of the paper are the baseline comparisons. There is a chapter for preliminaries that is referenced when describing what the proposed method is compared to, but only describes categories of different methodologies in a broad way. The actual description of the baselines and why they were chosen instead of others is too short or even missing making them hard to comprehend. It is especially unclear what part of the preliminary chapter is relevent for which baseline method, unless you are already familliar with them.

Thank you for your valuable feedback regarding our baseline comparisons and manuscript organization. We understand how important it is to ensure readability therefore, as it was a point raised by other reviewers, we kindly refer you to the general answer that notably takes into account both your relevant remarks.

Thank you for your thoughtful and detailed review of our submission. Your feedback has been valuable to guide our revisions.

We’ve worked to address the points you raised and believe these changes have improved the clarity and quality of the paper. As we haven’t yet received follow-up comments, we wanted to kindly check if there are any remaining areas where you feel further clarification or improvements would be beneficial.

We deeply value your insights and are eager to address any additional concerns to ensure the paper meets a suitable standard. We hope that the updates and improvements inspire confidence in our work, and we would be grateful if you would consider revisiting your evaluation.

Thank you again for your time and thoughtful contributions.

2. Clarifications and Revisions

.

We acknowledge the feedback from reviewers R#k3LF, R#Jdbp, and R#7hrj regarding certain aspects of our manuscript that could be improved. In response, we have made the following revisions (in blue in the resulting manuscript) :

.

Revisited Related Work Section (Section 2) :

• We condensed the discussion on Transfer Learning, Multitask Learning, and Meta-Learning into a single, concise paragraph. While these paradigms are foundational in Machine Learning, their relevance to Continual Reinforcement Learning (CRL) lies primarily in their overlap with certain methodologies. By simplifying this section, we maintain the necessary context without diverting from our focus on CRL.

• We also refined the categorization of CRL strategies, providing succinct descriptions of common and related approaches, along with their respective advantages and limitations. This refinement highlights the distinct challenges and contributions of our method within the broader CRL landscape.

• To better differentiate between Online and Offline CRL, we integrated discussions regarding their differences and the lack of suitable benchmarks. This integration underscores the lack of standardized benchmarks for Offline Goal-Conditioned CRL, further emphasizing the novelty of our framework. We also consolidated discussions about CSP and LoRA under a unified subsection and integrated Hierarchical Policies, ensuring a logical progression toward our contributions.

Enhanced Preliminaries Section (Section 3) :

• We enhanced the readability of the Preliminaries section by incorporating brief introductions and comments for key concepts. These additions guide readers to better understand how each component is used within, ensuring that the technical definitions are linked to their application in our framework.

• We clarified the role of Low-Rank Adaptation (LoRA) by detailing its concrete application in updating the subspaces. Specifically, we explain that new anchors within a subspace are generated using LoRA.

Methodology Section (Section 4) :

• We have clarified that Hierarchical Imitation Learning serves as the backbone when learning individual tasks within our framework.

• We have changed Section 4.2. title to explicitly show that it is a high-level overview of the algorithm before delving into detailed explanations. This reorganization ensures a logical progression and aids readers in comprehending the core learning steps involved in HILOW.

• We have included a brief explanation of why we use the Dirichlet distribution for sampling anchor weights during subspace exploration.