Thank you for your insightful review and for taking the time to engage critically with our work.

We have addressed each of your points in the responses below. If our clarifications resolve your concerns, we would be grateful if you would consider raising your score. Thanks again for your feedback, which has helped improve our work.

Weaknesses

Even though the proposed algorithm is a novel contribution to the best of my knowledge, its impact is debatable.

The approach can be extended to multi-task or multi-agent settings.

We want to emphasize that the contributions of our work lie not only in the technical components of the algorithm itself, such as the integration of HER with a goal-providing product MDP and Temporal Logic-derived minimum-safety constraints, but also in the accompanying theoretical analysis and empirical performance..

While the use of temporal logic in RL has seen progress over the past decade, we believe that key challenges still remain, particularly when scaling to continuous, sparse-reward, and safety-critical environments. Our method directly targets this space and advances the state of the art in integrating temporal logic with Deep RL.

We agree that extending this approach to multi-task and multi-agent settings would be a valuable next step. These directions pose non-trivial challenges and, in our view, our work provides a strong foundation upon which such work can be built.

Some important related work is missing [1, 2, 3, 4, 5, 6, 7].

Thank you for bringing these works to our attention. We agree that they are highly relevant and we will include them in the final version of the paper.

DeepLTL [1] is a relevant baseline, and the authors should definitely consider comparing against this work.

In terms of what is achieved, what are the differences between DeepLTL [1] and the proposed approach?

DeepLTL focuses on a different setting. It is designed for the multi-task paradigm, where the objective is to generalize across a distribution of LTL specifications using learned task embeddings. In contrast, our method targets single-task learning in high-dimensional continuous environments, with an explicit focus on safety.

Because of this difference, applying DeepLTL in our setting would require disabling several of its core mechanisms, such as multi-task generalization and curriculum-based training. For this reason, we believe a direct comparison would not be meaningful.

Additionally, DeepLTL places less emphasis on the RL backbone and on safety. It uses standard PPO within classical MDPs, whereas we adopt a CMDP formulation consistent with the safe RL literature, and introduce a product-CMDP construction along with a theoretical analysis.

From examining the experimental domains featured in both works, we believe a key distinction is our method’s ability to learn effective policies in environments with complex robot dynamics and explicit safety requirements. In particular, our use of minimum-safety constraints instead of rollout termination results in greater robustness and safety guarantees.

We view these two approaches as complementary. DeepLTL advances generalization across tasks, while our method focuses on safely and effectively satisfying a single temporal logic specification in complex continuous domains. Exploring how to combine the strengths of both approaches is a promising direction for future research.

A discussion of the difference between the proposed approach and [2, 3] is needed. Using HER in a multi-task RL context was proposed in [4].

Thank you for pointing this out. We will include these references and discuss them in the Related Work section of the final version of the paper.

Specifically, the work in [2] is also orthogonal to our work and provides a technique for providing additional rewards based on the agent’s partial completion of accepting cycles in the task automaton.

The work in [3] is also a multi-task RL approach that, similar to the LOF [8], trains an option for several goals and also conditions each option on a representation of the possible sequence of subsequent goals. [4] also is an interesting approach that uses HER directly with goals based on embeddings of task automata, which we believe is distinct from our method’s use of HER for densifying rewards in a single-task setting.

LTL formula embeddings [5] and automata embeddings [6, 7] can be used here instead of the subgoal lists in the product CMDP definition

What is the advantage of using subgoal lists over formula embeddings [5] or automata embeddings [6, 7]?

Thank you for this question. We agree that using LTL formula embeddings or automata embeddings is an exciting direction, particularly in the context of multi-task learning.

In the single-task setting, we find it more effective to rely on explicit subgoal lists. Compressing an entire temporal logic objective into a single embedding introduces a layer of abstraction that may obscure the precise semantics of the task, making learning harder and less interpretable. In contrast, subgoal lists offer a direct representation of the automaton's state and progression, which provides clear task guidance at each step.

Understanding how to combine the interpretability and task specificity of subgoal-based representations with the scalability and generalization potential of embedding-based approaches is an exciting direction for future work.

In line 133, it is said that the alphabet of the automaton is the MDP states. if the state space is continuous, then the alphabet is uncountably infinite.

We agree that defining a labeling function $L : \mathcal{S} \to 2^{AP}$ and the automaton in terms of the alphabet $\Sigma = 2^{AP}$, as opposed to directly defining the automaton transition function as $\delta: Q \times \Sigma \to Q$ with $\Sigma = \mathcal{S}$, will improve clarity and our consistency with the literature. We are considering accordingly modifying our definitions in the final version of the paper.

In line 137, It is unclear what is meant by propositional formulas on the transitions that hold for all symbols in the alphabet.

We introduce this definition of transition predicates, inspired by the terminology on page 3 of [9], to more easily discuss the implications of the automaton transition function $\delta$. These predicates still only are true for a subset of symbols, so it is not the case that any symbol can trigger any transition, and we believe this terminology is still compatible with Büchi automata.

In line 164, it is not clear what G^i is.

The $\mathcal{G}^i$ refers to the i-fold cartesian product of $\mathcal{G}$ with itself, and we agree this should be made more clear. We intend to resolve this in the final version for the paper.

It is not clear why subgoal lists facilitate the use of HER. Also, including Algorithm 1 in the main body of the paper might be helpful to the reader.

An obstacle to applying HER to a standard product MDP is that the standard product MDP defines no notion of goal that is compatible with the temporal logic task. HER would require such a goal in order to perform relabeling for the episode, but adding goal-conditioning while still supporting the task specified by the reward machine is not trivial. Other methods require introducing another representation of goals in order to enable goal-conditioning, such as the reach-avoid sequences in DeepLTL or automata embeddings in [4]. Our work instead introduces the subgoal lists to enable goal conditioning and HER for introducing additional rewards. Without this augmentation, HER cannot be easily applied in a standard product MDP.

We will do our best to include Algorithm 1 in the final version of the paper.

[8] Araki, Brandon, et al. "The logical options framework." International Conference on Machine Learning. PMLR, 2021.

[9] Alpern, Bowen, and Fred B. Schneider. "Recognizing safety and liveness." Distributed computing 2.3 (1987): 117-126.

Questions

What is the main reason behind the sim2real success?

The sim-to-real success can be attributed primarily to the close alignment between the simulation and the real-world environment. The simulated setup used a state space and action space that were not highly affected by sensor noise, matched the real robot’s dynamics, and accurately modeled physical constraints such as the shelf. Training and evaluation were carried out entirely in this simulator.

This type of sim-to-real transfer is common in policy learning when the simulation is carefully constructed to reflect the real-world system.

Have you tried any of the baselines in this setting? If so, what was their success rate?

Based on initial evaluation in the simulator, we were able to determine if trained policies were safe to deploy or not. The baselines did not perform well in the simulated environment, and therefore we did not believe it was appropriate to deploy them in the real world.

How do you pass the subgoal lists to the networks?

The subgoal list is processed by the networks according to how they appear in the boolean propositional statement $O(q)$ (defined in line 162) where $q$ is the agent’s current automaton state. Each goal is passed to a single-goal-conditioned value function module, and the output values for each goal are then combined according to min and max operations based on the “and” and “or” operators in $O(q)$ until a final value is obtained for the goal list. For more details, refer to the Appendix C Paragraph 1.

We greatly appreciate your thoughtful and positive review of our work. We have addressed the points you raised below and would be happy to provide any additional clarifications if needed. Thank you for your support.

Weaknesses

ACQL is currently restricted to Deterministic Büchi Automaton (DBA).

How do the authors envision extending their approach to more general STL/LTL classes or more complex automata?

We acknowledge that ACQL, like many automaton-based RL methods, currently focuses on tasks expressible by DBAs. However, we note that the majority of practically useful tasks, like those involving goal recurrence and safety, fall within this class. This is further supported by the fact that the most recent methods that are capable of handling more general specifications (e.g., those with persistence requirements) often do not demonstrate such tasks in their experiments [1]. This suggests the additional complexity they introduce remains a barrier in practice, and extending our approach to broader temporal logic classes remains a promising direction for future work.

We expect the first step in extending our approach will, similar to other methods, involve using non-deterministic Good-for-MDPs automata [2] (e.g., Limit-Deterministic Büchi Automata) to define our product CMDP and augmenting the action-space so that the policy may choose how to optimally resolve the resulting non-deterministic choices [3]. We are also interested in investigating how these non-deterministic choices are resolved in the presence of other reward signals and constraints.

it is not clear what they mean by stationary safety constraints.

in theory one could apply CMDP methods to the product CMDP

Thank you for raising this point. In this context, we mean by non-stationary that the safety constraints can change throughout a task as the agent progresses, as opposed to changing between learning and execution. It is possible to overcome this initial limitation by applying standard CMDP methods to the product CMDP, but we show in our second ablation (row 4 of Table 2) that standard sum-of-cost CMDP constraints struggle to address the safety requirements of our tasks compared to the minimum-safety formulation we propose.

the authors claim that the baselines "RMs and the LOF cannot be easily enhanced with high-performing GCRL and Safe RL techniques without adopting substantial modifications to their learning algorithms and product MDP formulation"

Thank you for the question. To clarify, applying HER to CRM-RS is not directly feasible, as HER requires a goal-conditioned agent, whereas CRM-RS operates in a standard product MDP that lacks an explicit goal representation for relabeling. Introducing goal-conditioning in such a way that does not modify the original temporal logic task is non-trivial; in our case, we achieve this by encoding each automaton state with its associated set of goals.

Similarly, LOF relies on a single product MDP and could, in principle, use HER to train a goal-conditioned policy in place of subgoal-specific options. However, this would then require adopting a different strategy for addressing the dynamic safety constraints imposed by the LTL specification, as each LOF option is trained both for reaching its associated goal and adhering to safety constraints relevant to that goal.

The author's mention STL, although all the formulas are provided with LTL notation, it is unclear how STL is used here

Thank you for raising this point. We specifically use the quantitative semantics (i.e., the robustness function) provided by STL in lines 216 - 219 to define the safety feedback function $c^A$. The remaining formulae are still expressed in STL notation, but without explicitly writing the default timing parameters ($[0, \infty]$) for each temporal operator.

Questions:

Could the authors compare ACQL conceptually to approaches that use shielding or model-checking at runtime (to prevent unsafe actions), or to other LTL-guided RL methods [6]?

These comparisons could be made as well. To summarize, shielding and model-checking for RL are, relative to our method, significantly more based on classic formal synthesis and verification techniques, and they consequently are more dependent on discrete models of the environment dynamics. Shielding specifically uses reactive synthesis to obtain, from a safety LTL specification and an abstraction of the system dynamics, a reactive system that can be used to minimally modify an arbitrary RL agent so that unsafe actions are never chosen. Runtime model checking approaches in general also rely on environment models to verify the overall safety of a policy and continuously check that runtime behavior matches the prediction of the model used in verification in order to ensure the policy remains safe. In contrast, our method does not rely on a model and supports more than just the safety fragment of LTL specifications.

Works such as [4] by Hasanbeig et al. on deep LTL RL are more comparable to our method, as they are model-free RL approaches relying on rewards from an automaton. However, we include the Reward Machines baseline as a similar algorithm that is slightly simpler and more appropriate for our fully-observable setting.

In practice does $\gamma_c \to 1$ introduce instability issues? How sensitive is ACQL to the scheduling of the safety discount factor and the learning rates on the two timescales? Could you explain how the contraction property can still be shown as $\gamma_c \to 1$?

In practice, this does indeed introduce some instability at values of $\gamma$ very close to 1, as small inaccuracies due to approximating the safety value function can quickly propagate and worsen the quality of the rest of the value function. Fortunately, this can be resolved by capping the $\gamma_c$ value at a maximum value of $0.99$. Because $\gamma_c$ only approaches and never equals or exceeds 1, the contraction property shown in Theorem 1 of [16] holds throughout the execution of our algorithm. On the other timescales, the learning rates for $Q^c$ and $Q^r$ did not have to be tuned at all for our algorithm to succeed in practice.

Further, did you have to tune the rate at which $\gamma_c$ approaches 1, or did a simple schedule (e.g., linear annealing) work out-of-the-box?

Very minor tuning was initially required so that $\gamma_c$ increased slowly enough so that an initial safety value function could be quickly learned, but after this a straightforward schedule based on exponential decay worked out-of-the-box.

Does the training process itself ever cause safety violations in the environment? If the agent takes an unsafe action, is the episode terminated or is it allowed to continue with a cost? In real-robot experiments, how were safety violations handled to protect the hardware?

Our algorithm does require experiencing transitions that violate safety requirements in order to learn the safety value function, and so we do not terminate episodes upon safety violations. We note that a buffer can be instated for the safety requirements so that safety violations do not immediately result in harm to the robot.

For our real-robot experiments, training was exclusively performed in simulation to the point that safety constraints were not encountered during deployment of the policy.

How can ACQL be adapted to partially observable settings?

We wish to clarify that a partially observable setting in this context refers to a domain where the agent can only receive partial observations (e.g. images) or imperfect information of the underlying system state, but the task specification is still defined in terms of this underlying state.

In order to extend ACQL to this setting, we envision first augmenting the network so that throughout execution the history of observations is processed to obtain a latent representation of the underlying state, with which a Markovian policy should be well-defined. Second, because atomic propositions relevant to the task specification can not be directly measured but instead must be deduced from the history of observations, our extended method should assume access to or learn a probabilistic function, similar to the one used in [4], for evaluating propositions.

Subsequently, our method could maintain a probability distribution over automaton states to represent the agent’s belief about its current progress and update this distribution based on the incoming observations. Periodically sampling from this distribution would then yield a concrete subgoal list, which will be relabelled based on the final achieved subgoals. We believe this to be a very interesting and challenging direction for future developments of our work and the field of policy learning methods using temporal logic.

[1] Jackermeier, Mathias, and Alessandro Abate. "Deepltl: Learning to efficiently satisfy complex ltl specifications for multi-task rl." The Thirteenth International Conference on Learning Representations. 2025.

[2] Hahn, Ernst Moritz, et al. "Good-for-MDPs automata for probabilistic analysis and reinforcement learning." International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Cham: Springer International Publishing, 2020.

[3] Hahn, Ernst Moritz, et al. "Omega-regular objectives in model-free reinforcement learning." International conference on tools and algorithms for the construction and analysis of systems. Cham: Springer International Publishing, 2019.

[4] Hasanbeig, Mohammadhosein, et al. "Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees." 2019 IEEE 58th conference on decision and control (CDC). IEEE, 2019.

Thank you for taking the time to review our work and for your detailed feedback. We have addressed each of your comments and clarified the points that may have been unclear in the original submission. If our responses resolve your concerns, we would sincerely appreciate your consideration in updating your score. If there are any remaining questions, we would be glad to provide further clarification.

Weaknesses

While there is value in the proposed approach, the overall contribution is not substantial enough to constitute a full paper at a conference. Trying to address some of the limitations highlighted in the end would benefit the overall impact of the paper.

Thank you for recognizing the value of our approach and for raising this point. We want to emphasize that the contributions of our work lie not only in the technical components, such as the integration of HER with a goal-providing product MDP and Temporal Logic-derived minimum-safety constraints, but also in the accompanying theoretical analysis and practical performance.

These elements together offer a principled and scalable framework for learning to satisfy singular temporal logic specifications in high-dimensional continuous state spaces. We agree that addressing the remaining limitations would represent an important next step. However, these challenges are not unique to our method and are common to many approaches in the field. Our work takes meaningful steps in this direction and, in our view, provides a valuable foundation that the community can build upon.

An analysis of the effect of the complexity of the task on the sample complexity is missing. It is particularly important as the observation space is expanded with the set of sub-goals.

We would like to clarify that the observation space is not necessarily expanded as the number of subgoals increases. As described in lines 172-176, our algorithm augments the observation only with the goal corresponding to the next relevant automaton state. When the automaton includes branching, we concatenate all possible successor automaton states to capture the multiple paths forward. However, in the absence of branching, only a single goal is included. This design keeps the observation space compact in the majority of practical scenarios.

While it is theoretically possible for the observation space to grow with automaton size in tasks with extreme branching, we believe this is rare in the types of robotic applications and temporal objectives we are targeting.

We agree that a deeper investigation into the connection between automaton complexity and sample complexity would be valuable. However, given our current scope, we believe the most immediate contributions lie in demonstrating the practical feasibility of Temporal-Logic-based learning in continuous, high-dimensional environments.

The authors state that the minimal safety constraint is better over the discounted sum of cost.

Thank you for the question. We believe that using the minimal safety constraint provides advantages over the discounted sum of costs in almost all cases. We interpret hard safety as referring to constraints that cannot tolerate any violation, while soft safety allows for partial violations. Our approach can accommodate both by adjusting the threshold $L$ in Equation 2, in much the same way as one would tune $L$ in Equation 1 for the discounted cost formulation.

The key benefit of our approach lies in the learning feasibility. Training a network to directly classify safety (i.e., learning the sign of $\min_{t \in [0, \infty]} c^A_t$) is significantly easier and more stable than learning to predict a discounted sum of costs and then thresholding the result. For evidence, consider the lower-bounds on sample-complexity for learning a Probably-Approximately Correct binary classifier given in Theorem 6.8 of [1] and for a real-valued regression model given in Theorem 2.1 in [2]. Additionally, the threshold $L$ in Equation 2 has a more direct interpretation as the maximum allowable safety violation, rather than the total accumulated cost, which is harder to reason about in safety-critical applications.

We will make sure to clarify these points in the final version of the paper.

[1] Shalev-Shwartz, Shai, and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.

[2] Morgenstern, Jamie H., and Tim Roughgarden. "On the pseudo-dimension of nearly optimal auctions." Advances in Neural Information Processing Systems 28 (2015).

The authors decided to not use any of the Safe RL methods as baseline with the justification that "Safe RL methods typically address stationary safety constraints". Only one of the tasks features such non-stationarity.

We agree that the current justification in the paper is not sufficiently clear, and we will revise the quoted sentence to provide a more explicit and well-motivated explanation.

Specifically, while Safe RL methods typically focus on stationary constraints, their scope is also fundamentally different from ours. They are not designed to satisfy general Temporal Logic specifications.

We will clarify this reasoning in the final version of the paper, around lines 268 to 270.

It is not entirely clear to me whether extending CRM-RS with HER will not achieve the same result.

We would like to clarify that simply extending CRM-RS with HER would not achieve the same result as our proposed method. While it is possible in principle to augment CRM-RS with HER, doing so is not straightforward. HER requires a goal-conditioned agent, and adapting CRM-RS to support this would require explicitly constructing a goal-providing product MDP.

Even if such a combination were implemented, it would still rely on rollout termination to enforce safety constraints. Our experimental results show that this approach is significantly less effective than the minimum-safety constraint we introduce.

It is hard to believe that RMs, especially on 1st and 2nd task in the table, almost never solve the tasks.

While reward machines are indeed well-suited for representing temporal logic tasks, their effectiveness has primarily been demonstrated in relatively small or discrete environments where rewards are encountered frequently and value function learning progresses more easily. In contrast, the environments we consider are significantly more challenging. They involve high-dimensional continuous state spaces and require long sequences of actions to reach even a single reward, resulting in extremely sparse reward signals.

In such settings, standard reward-machine-based methods struggle to make meaningful progress, as the agent rarely observes positive feedback. This motivates our proposed approach using goal relabeling and safety-aware learning. Our results reflect this difficulty and highlight the necessity of the design choices made in our method.

Writing Issues

Wrong citation for GCRL [1]

We believe the Universal Value Function Approximators work is an appropriate citation, as it represents a starting point for the field of deep goal-conditioned RL.

line 232: should it not be in $\mathcal{S}^A$

formula (3) - elements such as $r_t$ seem to come from the batch but it is not explicitly specified

While the states $s_t^A$ in Line 232 are also members of $\mathcal{S}^A$, we meant by writing $s_t^A \in \mathcal{B}_i$ that these are the states included in the batch of sampled transitions $\mathcal{B}_i$ used for computing the loss in Equation 3. We will further clarify this notation near Equation 3 in the final version of the paper.

Questions:

What would you think of comparing the method against offline policy learning? It is a fundamentally different problem?

We agree that combining offline policy learning with temporal logic objectives is an interesting and promising direction for future research. For instance, one could imagine collecting offline data using policies that satisfy certain LTL specifications and then training new policies to generalize to unseen temporal tasks. However, this would involve a different problem formulation, with different assumptions, training regimes, and evaluation goals.

Our focus in this work is on online policy learning with sparse feedback and safety constraints, where the agent must learn from limited, task-driven interaction in high-dimensional continuous spaces. Exploring how offline learning techniques could complement or extend our framework is a natural and exciting next step, but beyond the scope of the current study.

The labeling function might be undefined. The use of Signal Temporal Logic and $\rho$ is unmotivated.

In our formalism, we chose to define the automaton transition function directly as $\delta: \mathcal{Q} \times \Sigma \to \mathcal{Q}$ with $\Sigma = \mathcal{S}$, rather than introducing an explicit labeling function. This can be equivalently viewed as the composition of a labeling function $L: \mathcal{S} \to 2^{AP}$ and a transition function defined over $2^{AP}$. While this choice simplifies notation in our setting, we agree that including the labeling function explicitly would improve consistency with standard treatments of automata-based RL and with prior work such as [8]. We are considering adding this to the final version of the paper for completeness and clarity.

The use of Signal Temporal Logic (STL) in our framework is still compatible with a labeling function. Each atomic proposition $p \in AP$ is associated with a function over states, and the labeling can be constructed based on their truth values. The robustness function $\rho$ provided by STL offers an additional advantage by enabling a principled and differentiable way to define our safety constraint function $c^A$, as introduced in line 217.

Thank you very much for taking the time to review our work and for your positive evaluation. We have addressed the points you raised below and would be happy to provide any additional clarifications if needed. We sincerely appreciate your feedback and support.

Weaknesses

It's unclear whether experimental setup considers deterministic or stochastic environments.

Thank you for highlighting this point. Our current experiments are indeed implemented using a deterministic simulator, and we agree that this detail should be clarified more explicitly in the final version of the paper. That said, we would like to emphasize that our algorithm is formulated without any assumption on deterministic dynamics and is fully compatible with stochastic environments.

In obtaining the Automaton... paragraph, it appears that proposition identification is manual. If so, how would misidentification of these subgoals affect the results?

We confirm that identification of subgoal propositions is currently performed manually, as described in lines 176-178. However, this process is straightforward and involves selecting atomic propositions that correspond to goal-reaching behaviors specified in the given temporal logic formula. These are typically the propositions that encode states the agent must eventually reach.

While it is possible to automate this step in general, we found that manual identification is reliable and poses a minimal one-time overhead in practice. Misidentification would only occur in the case of a malformed or ambiguous specification, in which case any high-level reasoning algorithm would likely fail to generate meaningful goals. In our experiments, we encountered no such issues.