Imitation Learning with Temporal Logic Constraints

We appreciate your insightful feedback and constructive comments!

Q1.1 Why would segmentation only benefit tasks without absorbing accepting states?

In tasks with absorbing accepting states (e.g., Doggo, Fetch), the agent is expected to remain in the accepting state once it is reached, leaving no further progression to segment or imitate—aside from maintaining safety (e.g., avoiding sink states). In such cases, segmentation reduces to a trivial one-step segment exactly at accepting states and primarily serves to provide safety-related reward signals. Segmentation is particularly beneficial for tasks without absorbing accepting states, where the agent must repeatedly visit, leave, and return to accepting states. These tasks exhibit rich, multi-phase behaviors, and segmented imitation enables TiLoIL to learn policies that cycle between accepting states—effectively supporting repeated satisfaction of the LTL objective. We emphasize that even in tasks with absorbing accepting states, TiLoIL still benefits from multi-stage discriminator learning and reward formulation, as shown in Figure 4.

Q1.2 Following lines 187-188, does this have something to do with satisfying LTL constraints over infinite horizons being more challenging? Why?

Lines 187–188 highlight the challenge of imitation learning under LTL constraints: the agent must generalize from finite-length demonstrations to satisfying infinite-horizon objectives. In practice, user-provided demonstrations may be suboptimal due to the cost and complexity of constructing trajectories that robustly satisfy such objectives. In our setup, demonstrations are generated by chaining locally trained controllers for each stage of the LTL task (line 341). These policies are not globally optimal: when executed repeatedly, they often fail to produce trajectories that visit LDBA-accepting states infinitely often. For example, in CheetahFlip, the chained controllers—and thus the resulting demonstrations—achieve only one or two rotations, since additional flips require maintaining momentum after reaching an accepting state, which the learned local policies cannot sustain. Similarly, in Carlo, demonstrations rarely reach accepting states more than twice. Despite this suboptimality, TiLoIL learns policies that outperform the demonstrations, empirically visiting accepting states more frequently in both CheetahFlip and Carlo.

Q1.3 Why motivating the use of infinite-horizon tasks but then most of the tasks are finite-horizon? (Please let me know I am wrong in saying that tasks with absorbing accepting states do not have an infinite horizon.)

While several tasks in our benchmark suite include absorbing accepting states, they are still formally infinite-horizon: the agent operates in an unbounded setting where episodes do not terminate upon reaching accepting states. Crucially, the agent must continue to satisfy any ongoing safety properties—e.g., avoiding unsafe regions in Doggo Avoid—even after accepting states are reached. Importantly, 5 out of 9 tasks in our suite require policies with meaningful $\omega$-regular structure to satisfy the specification—for example, visiting certain regions repeatedly. These tasks truly exhibit the types of behaviors that infinite-horizon LTL specifications are designed to express. We will clarify the distinction between infinite-horizon formulations and absorbing structures in the revised version.

Q1.4 Why are the ablations not performed (or shown) for GridCircular Hard or FlatWorld tasks?

The ablations for GridCircular Hard and FlatWorld are shown below. Both multi-stage reward formulation and segmented imitation are important. Performances are measured by eventual discounted returns over 10 seeds.

Q2. In what contexts would finite-length demonstrations be potentially problematic or insufficient? There are infinite-horizon tasks where the environment can change, e.g. see the tasks Cookie and Symbol in [1]. Under which circumstances would demonstrations in these settings be useful? Have any inherent assumptions on the demonstrations been made in the submitted work? Is the assumption visiting an accepting state only once or twice stronger than it seems?

A finite number of demonstrations may become insufficient in more dynamic settings, such as those with varying accepting conditions, if they fail to capture the necessary behavioral diversity.

To evaluate TiLoIL in such settings, we adapted two environments inspired by Cookie and Symbol from [1].

• Cookie-like task: We modified the Fetch Pick&Place environment by introducing a button on the table. Pressing the button causes a goal region to appear. The manipulator must (1) press the button, (2) pick up a block and move it to the goal region, and then (3) repeat this sequence with the goal region appearing in a new location each time. Compared to the baselines, TiLoIL was the only method that successfully solved this task within 2 million steps, demonstrating its ability to handle changing environments.

• Symbol-like task: We redesigned the original Symbol task into the Pick&Place environment. The workspace contains a movable block and two potential destination regions: a tray and a goal region. A tray may or may not appear in the scene at each episode. The symbolic cue is the presence or absence of the tray:

  • If the tray appears, the agent must place the block into the tray.
  • If the tray is absent, the block should be placed into the goal region.

The LTL specification is (tray_present -> F in_tray) && (Not tray_present -> F in_goal). This task highlights a core challenge in learning from partial demonstrations. When demonstrations only show placing the block into the goal region (i.e., when the tray is absent), TiLoIL fails to generalize to the unseen tray-present case. It always moves the block to the goal region regardless of tray appearance. However, when demonstrations cover both cases (tray present and tray absent), TiLoIL successfully learns the correct conditional behavior.

These experiments clarify our assumption: demonstrations must sufficiently cover the behavioral variations necessary to reach the accepting conditions under different modes. When demonstrations cover only a subset of the possible cases, the learned policy may become biased toward that subset and fail to generalize. We will include a clearer articulation of this assumption, along with a discussion of these experiments, in the revised manuscript. We will clarify that “reaching accepting states once or twice” is not a criterion for demonstration quality. Rather, this was a practical design choice in our experimental setup to showcase that TiLoIL can still succeed even when demonstrations are sparse for the benchmarks we considered.

Q3. Has the effect of having longer or shorter demonstrations (hence, reaching the accepting states more or fewer times) been studied? If not, how do you think it can influence the performance of the algorithm?

The table below illustrates the impact of demonstration length on performance. Performances are measured by eventual discounted returns over 10 seeds. The first column uses demonstrations of 100 steps, while the second and third columns extend this to 200 and 400 steps, respectively. In the Carlo environment, longer demonstrations improve learning outcomes. For CheetahFlip, extending the demonstration length does not help because the demonstrations are suboptimal (see Q1.2) and longer trajectories do not provide more visits to accepting states.

Q4.1 Has the approach been compared against methods that leverage the structure of an automaton for reward shaping?

Indeed, earlier works by Camacho et al. [12, 13] propose potential-based reward shaping using distances in product MDPs (based on the minimum number of steps to accepting automaton states). Another work [31] extend this idea by computing potential functions via value iteration on a synthesized MDP from the automaton of a temporal property. These approaches are not directly applicable to ω-regular problems. By adopting an LDBA instead of a DFA, and integrating an eventual discounting scheme, more recent approaches introduce reward shaping methods tailored for the full expressiveness of LTL. For example, Bagatella et al. [6] (DRL2) propose a Bayesian method that models LDBAs as Markov reward processes with learned transition kernels, enabling potential-based shaping via high-level value estimation. Similarly, Shah et al. [55] (Cycler) distribute heuristic rewards along LDBA accepting paths. In our paper, we compare TiLoIL against DRL2 and Cycler as the most relevant potential-based reward shaping baselines. As shown in Fig. 3, TiLoIL consistently outperforms both, demonstrating the benefit of leveraging demonstration guidance.

Q4.2 How about method using hierarchical reinforcement learning, which similarly leverage the decomposition of a task into substages?

Indeed, several works within the LTL literature leverage the structure of automata to define hierarchical policies, such as Hasanbeig et al. (2020), Icarte et al. (2022), and Jothimurugan et al. (2021) [28, 31, 35]. These methods decompose the overall task into subtasks associated with automaton transitions, improving sample efficiency. However, a known limitation of such hierarchical approaches is potential suboptimality due to local policy decisions at each subtask level [31].

I appreciate the authors' responses, which have positively addressed all my concerns; hence, I will likely increase my score just before the end of the discussion period.

In summary, the paper would highly benefit from the following in future versions:

• Clarifying the distinction that tasks with absorbing accepting states are still infinite-horizon (related to Q1.1-Q1.3).
• Including the ablations in Q1.4 (it could be in the appendix if there is no space).
• Including the adapted environments from Q2, Cookie-like task, and Symbol-like task, adds insight into the potential and the limitations of the approach. Including them in the main paper (or adding them to the appendix, with summarized findings in the main paper) would really help drive future research in the field. Adding the related assumption clarification would be of great value.
• The analysis on the demonstration length (Q3) is also interesting because it shows that longer demonstrations do not always improve performance.
• Restructure the closing of the paper (conclusions and future work should not be within the related work).

By reading the authors' answers to other reviewers, the authors should be clearer about the contribution (i.e., learning from finite-length non-optimal demonstrations), the challenges in the scenario they tackle (e.g., not necessarily optimal demonstrations), how it relates to previous work (e.g., inverse RL, which may help emphasize the possible absence of optimal demonstrations), and the assumptions on the demonstrations (also tackled in the response to this review).

We appreciate your insightful feedback and constructive comments!

Limitation: The proposed method faces scalability challenges due to its multi-stage discriminator learning framework. Specifically, the number of accepting states grows with the complexity of the LTL specification and the number of atomic propositions, leading to a corresponding increase in the number of discriminators. This scaling makes the training process more computationally demanding. Moreover, jointly training multiple discriminators alongside the Q-function and policy introduces additional optimization difficulties and may hinder stable convergence.

LDBAs are commonly used for LTL-to-automaton translation because they yield automata significantly smaller than deterministic Rabin or full deterministic automata, keeping the automaton size manageable while preserving the structure needed for verification and learning. Particularly, the number of accepting states in an LDBA is small—often just one in our benchmarks—and does not scale with specification complexity. In practice, all benchmark LTL specifications in our work (and related studies) produce LDBAs with ≤ 10 states, resulting in only a small number of discriminators and computationally feasible training. We acknowledge that jointly training multiple components can raise stability concerns; however, TiLoIL exhibited stable convergence in all experiments (see Figs. 3–4) with no instability beyond what is typical in adversarial imitation learning (e.g., GAIL). Our framework can be extended to share parameters across discriminators or use belief-weighted mixtures, which would reduce the number of separate networks needed.

Q1. In Eq. (4), the reward is defined as a function of the product state $R((s,b))$, independent of the action $a$. This differs from the standard RL setting, where the reward typically depends on both the state and the action, i.e., $R(s, a)$. Could the authors clarify the rationale behind this formulation and its implications?

Our formulation follows standard practice in automata-guided RL and formal methods, where the objective is to find $\pi^\star \in \arg\max_{\pi\in\Pi} P(\pi\models\varphi)$, the policy that maximizes the probability of satisfying an LTL specification $\varphi$. This probability can be written as the expected value of an indicator over trajectories satisfying $\varphi$ (Line 120). Therefore, the rewards need only encode progress toward specification satisfaction (i.e. reaching an LDBA-accepting state), which is determined solely by the product state $(s,b)$ and not by the specific action taken. On the other hand, although $R((s,b))$ does not directly distinguish between actions, the transition dynamics do: different actions from the same state lead to different future product states, resulting in different expected cumulative rewards. Consequently, the optimal policy remains well-defined, and action choices still affect performance through their impact on future specification progress.

Q2.1 For the multi-stage reward formulation, how does the choice of the hyperparameter $\beta$ affect performance, given that the reward directly influences policy learning?

In our reward formulation, $\beta$ controls how much the shaped rewards from multi-stage discriminators weigh. All the experiments in the paper are done by $\beta = 1/3$, the table below shows how different values of $\beta: (0.49, 1/3, 1/4)$ affect performance across environments, measured by eventual discounted returns over 10 seeds.

All our ablations outperform the baselines. For easy tasks like GridCircular Hard and FlatWorld Stabilization, this hyperparameter doesn't have a significant impact. Some challenging tasks such as Fetch Align and CheetahFlip perform better with larger $\beta$, meaning the shaped reward from the discriminator guides the learning more effectively.

Q2.2 Additionally, since the reward depends on the length of the longest acyclic path, could the authors provide more details on how this is computed and how it impacts learning, especially in cases where the path is long, which may lead to very small or vanishing reward signals?

We compute the longest acyclic path by performing depth-first search to enumerate all simple paths from a given automaton state to any accepting state. If a state is encountered that is already on the current path, we do not extend the path further through that state. Since LDBAs are typically small and structurally constrained in our benchmarks, this approach remains computationally tractable in all tasks we considered. Normalizing the progress-based reward in Eq. 8 by the length of the longest path is beneficial for stabilizing training as it ensures that all progress signals are kept within a consistent [0,1] range, which helps maintain reward sensitivity throughout training. Empirically, we did not observe vanishing gradient issues—likely due to the sparse nature of LDBAs.

Q3. The proposed method assumes access to expert demonstrations. Could the authors clarify the underlying assumptions about these demonstrations? Specifically, do the demonstrations contain trajectories that visit each accepting state, and do they provide sufficient coverage of the state space? For instance, in the FlatWorld Cycle environment, if the demonstrations only reach the yellow region, how would the method support learning policies that reach the green region? Similarly, if all demonstrations originate from the left side of the environment, would this limit the agent’s ability to explore or learn effectively when initialized from the right side?

• TiLoIL assumes that the demonstrations cover key transitions of an LDBA, especially by visiting accepting states at least once or twice (Line 184), thereby enabling it to identify and learn from successful automaton progressions. We provide a detailed discussion and examples of this assumption in our response to Reviewer Tszg (Q2).

• However, TiLoIL does not require the demonstrations to provide sufficient coverage of the state space because it is able to leverage the agent's own behaviors—those that have progressed beyond certain LDBA states—as positive trajectories to train discriminators that guide learning for those trajectories that fail to meet such LDBA states, thereby reducing reliance on large demonstration datasets (Fig. 4 further supports this, as TiLoIL_noStage, which lacks multi-stage discriminators and resembles GAIL, performs worse). In FlatWorld Cycle, if we use the demonstrations that originate from the left side of the environment, and train the agent starting from the right, it still learns well (converging to 3 returns after 200K steps).

• Importantly, TiLoIL does not require user-demonstrations to be of high quality. In practice, user-provided demonstrations may be suboptimal due to the cost and complexity of constructing trajectories that robustly satisfy such objectives. In our setup, demonstrations are generated by chaining locally trained controllers for each stage of the LTL task (line 341). These policies are not globally optimal: when executed repeatedly, they often fail to produce trajectories that visit LDBA-accepting states infinitely often. For example, in CheetahFlip, the chained controllers—and thus the resulting demonstrations—achieve only one or two rotations, since additional flips require maintaining momentum after reaching an accepting state, which the learned local policies cannot sustain. Similarly, in Carlo, demonstrations rarely reach accepting states more than twice. Despite this suboptimality, TiLoIL learns policies that outperform the demonstrations, empirically visiting accepting states more frequently in both CheetahFlip and Carlo.

Q4. In the experiments, how many episodes are averaged per evaluation (in addition to the 10 random seeds) for each environment? Also, aside from Cheetah Flip, are the initial positions of the agents and the placement of regions or objects randomly generated during evaluation?

We evaluated each policy using 10 episodes per random seed, averaged across 10 seeds. The initial positions are random in Doggo, Fetch, and Carlo environments; fixed in FlatWorld environments.

We appreciate your insightful feedback and constructive comments!

Q1. What is the effect of adding non-optimal demonstrations in practice? Will the results degrade gracefully, or can one sub-optimal demonstration, no matter how close to satisfaction, collapse the entire optimization?

We clarify that the assumption of expert demonstrations applies only to our theoretical analysis (e.g., Theorem 3.1). In practice (line 341), our demonstrations are generated by chaining locally trained controllers for each stage of the LTL task. The policies learned in this way are not optimal: when executed repeatedly, they fail to produce trajectories that visit LDBA-accepting states infinitely often. For example, in CheetahFlip, the chained controllers—and thus the collected demonstrations—achieve only one or two rotations, since additional flips require maintaining momentum after reaching an accepting state, which the learned local policies cannot sustain. Similarly, in Carlo, demonstrations rarely reach accepting states more than twice. Despite this suboptimality, TiLoIL learns policies that surpass the demonstrations, empirically achieving higher visitation rates of accepting states in both CheetahFlip and Carlo. A key reason for this is that TiLoIL does not attempt to imitate the demonstrator's actions—in fact, we assume that demonstrations consist only of state trajectories (Line 138). This allows TiLoIL to prioritize task satisfaction over action mimicry, leveraging the logical structure of the LTL objective to generalize beyond the demonstrations.

Q2. If the epsilon-transitions have any meaning in your framework, I'd love to see them incorporated into an example.

We will revise the paper to include the FlatWorld Stabilization environment as an illustrative example to clarify the role of ε-transitions. In this environment, the LTL task is FG $y$, which requires the agent to eventually reach and then remain in the yellow region. The corresponding LDBA consists of three states. The transition from automaton state 0 to state 1 is an ε-transition, which occurs independently of the MDP state—it represents the logical satisfaction of the "finally" operator in the LTL formula rather than any concrete observation. Once in state 1, the automaton remains there as long as the agent stays in MDP states satisfying $y$ (i.e., remaining in the yellow region). If $y$ becomes false, the automaton transitions to sink state 2. The learned policy must learn to initiate the ε-transition (Def. 2.3) at the right time and then enter the yellow region, taking actions that ensure the agent stays there indefinitely and avoiding the sink state. This illustrates how ε-transitions enable abstract progression in the automaton that is not tied to any specific environment transition, yet must be orchestrated carefully by the policy in concert with the environment dynamics.

Q3. Can this method be adapted to a setting where the observations of the environment are noisy/partially observed? What if I'm only 90% certain I hit 'blue'?

Yes! Thanks for the suggestion. Our method can be extended to handle models uncertainty in atomic propositions via belief-based policy learning. In the POMDP setting (environment are noisy/partially observed), the agent has only access an observation distribution $\omega : \mathcal{S} \rightarrow \Delta \mathcal{O}$ induced by the state space $\mathcal{S}$ - the agent observes an observation $o_t \sim \omega(s_t)$ and performs an action based on $o_t$ instead of $s_t$. Denote the agent's observation-action history at time $t$ by $h_t = (o_1, a_1, \ldots, o_{t-1}, a_{t-1}, o_t)$ and the set of all possible histories as $\mathcal{H}$. In principle we can apply any deep RL approach suitable for POMDPs, such as a recurrent policy $a_t \sim \pi(a_t \mid o_t, b_t)$ that conditions on $h_t$ and the current LDBA state $b_t$. However, we don't have access to the ground-truth labeling function $\mathcal{L}$ and therefore cannot accurately evaluate $b_t$. As pointed out by the reviewer, we can assume we access to a model $M : \mathcal{H} \rightarrow \Delta(2^{\mathsf{AP}})$ that predicts probabilities over possible propositional evaluations $\mathcal{L}$ (e.g. 90% certain agent hitting 'blue'). We therefore can use $M$ to predict a distribution over LDBA states $\tilde{b}_t \in \Delta \mathcal{B}$:

• At $t=0$: we initialize the belief \tilde{b}_0\ to be certain of the initial LDBA state $b_0$.
• For $t > 0$: we propagate belief forward through possible transitions ${P}^\mathcal{B}$, using uncertain information about the environment from $M$:

The probability that the LDBA is in state $b$ at time $t$ is computed by summing over all possible prior LDBA states $b'$ and all possible atomic proposition sets $\Sigma \cup \mathcal{E}$, and weighing:

• whether P^\mathcal{B}(b', \sigma) = b\ (i.e., whether the LDBA would transition from $b'$ to $b$ if evaluation $\sigma$ to atomic propositions were observed),
• how likely it was that we were in state $b'$ at $t-1$ (i.e., $\tilde{b}_{t-1}\[b']$),
• how likely it is that $\sigma$ is the set of atomic propositions true at time $t$ (from $M(h_t)[\sigma]$).

The decision-making module is a policy $\pi(a_t \mid h_t, \tilde{b}_t)$ that then leverages the inferred belief $\tilde{b}_t$.

At each time step $t$, we can then compute a soft multi-stage reward from the bank of discriminators: $r_t = \sum_{b \in \mathcal{B}} \tilde{b}_t[b] \cdot R((s_t, b))$ for training the agent, where the reward function $R$ is our original multistage reward formulation defined for a product MDP state in Equation (8), Line 302. In this way, $r_t$ smooths out errors in uncertainty in atomic proposition truth values.

To train each stage-specific discriminator $f_b$ (Line 293), we can similarly compute the belief of a trajectory that has passed LDBA states, storing the trajectories in stage buffer with soft weighting — e.g., weight a sample by $\tilde{b}_t[b]$ when training $f_b$.

We see the integration of our reward shaping into both multi-task frameworks and noisy environments as exciting directions for future work.

Weakness: The contributions of the paper are not clearly outlined.

We appreciate this feedback. Our primary contribution is to show that finite-length demonstrations can be used to overcome the exploration bottleneck inherent in learning for LTL objectives over infinite horizons. We will revise the introduction to include an explicit “Contributions” section that makes our key contributions more visible to readers.

We thank the reviewer for the helpful clarification.

I was thinking of a different type of suboptimality – demonstrations that violate the formula by failing to hit some sub-goals, for example.

We agree that such demonstrations are particularly challenging. However, our method is designed to degrade gracefully in their presence—suboptimal demonstrations do not collapse the optimization.

As described in Line 5 of Algorithm 1, we still assign demonstration trajectories to LDBA accepting sets $\mathcal{B}^\star$, even if they fail to fully complete the task. Demonstrations that fall short of fully satisfying the LTL formula—such as those that progress through some sub-goals or approach accepting states without fully reaching them—can still offer valuable learning signals during training.

To further validate this, we conducted an additional experiment in the Carlo environment. This environment involves a self-driving agent trained to follow a circular track counterclockwise while visiting two blue regions repeatedly and avoiding crashes. We trained TiLoIL using only failed or incomplete demonstrations, including those that either entered unsafe regions or failed to complete a full loop by missing one blue region. Results (averaged over 5 seeds after 400K training steps) are shown below:

While there is a performance drop—as expected—TiLoIL remains effective, significantly outperforming the RL for LTL baselines and GAIL, as reported in the main paper. We will include this discussion and the results in the revised version to clarify the method's robustness under such suboptimal demonstrations.

In the revision, as discussed in our initial response, we will add the FlatWorld Stabilization environment as an illustrative example to clarify the role of ε-transitions, discuss how TiLoIL can be adapted to environments with noisy or partial observations, and revise the introduction to include an explicit “Contributions” section. This section highlights our main contribution: demonstrations from suboptimal agents that can only reach LDBA-accepting states a limited number of times (e.g., once or twice in our experiments) can be effectively leveraged to learn policies that satisfy LTL objectives over infinite horizons. We appreciate your comments and suggestions, which strengthen the clarity and scope of our work!

We appreciate your insightful feedback and constructive comments!

Q1. Are there any implicit objectives in the demonstrations beyond the LTL specifications?

W1. Why don't all expert demonstration satisfy the given LTL? The setup is presented as multi-objective, but it seems the primary goal is to find a policy that satisfies LTL constraints, unless there are additional implicit objectives encoded in the demonstrations.

Our approach (TiLoIL) is developed for Imitation Learning with Temporal Logic Constraints. For imitation learning, one can simply optimize $\pi^\ast = \arg \max_{\pi_\phi \in \Pi} J_\pi(\phi)$ where $J_\pi(\phi)$ is the GAIL objective minimizing the Jensen-Shannon) divergence between the learning policy and expert policy (Line 159). However, in practice, user-provided demonstrations may be suboptimal due to the cost and complexity of constructing them. Learning solely from finite-length demonstrations does not guarantee deriving a policy that generalizes to infinite horizon LTL tasks. As a concrete example, in our setup, demonstrations are generated by chaining locally trained controllers for each stage of the LTL task (line 341). These policies are not globally optimal: when executed repeatedly, they often fail to produce trajectories that visit LDBA-accepting states infinitely often. In CheetahFlip, the chained controllers—and thus the resulting demonstrations—achieve only one or two rotations, since additional flips require maintaining momentum after reaching an accepting state, which the learned local policies cannot sustain. Similarly, in Carlo, demonstrations rarely reach accepting states more than twice. To address this issue, we add LTL constraints forming a multi-objective learning paradigm: $\pi^\ast = \arg \max_{\pi_\phi \in \Pi} \left( P(\pi_\phi \models \varphi), J_\pi(\phi) \right)$. The agent not only imitates demonstrations but also maximizes the probability of LTL satisfaction. Under this formalism, despite demonstration suboptimality, TiLoIL learns policies that outperform the demonstrations, empirically visiting accepting states more frequently in both CheetahFlip and Carlo.

Although demonstrations do not encode implicit objectives, the multi-objective formulation—combining demonstrations with LTL specifications—provides a compelling advantage: it enables policies to be learned significantly faster than with pure RL, while requiring far fewer demonstrations than classical imitation learning.

Q2. How should TiLoIL be categorized in relation to existing work (e.g., online vs. offline, imitation learning vs. reinforcement learning, or a hybrid)?

TiLoIL is a hybrid online imitation learning method. It extends the adversarial imitation learning framework of GAIL to tasks specified by LTL objectives. Like GAIL, it performs online interaction with the environment while training a discriminator on expert demonstrations to provide a shaped reward signal. The demonstrations are used to guide exploration in the early stages, but the policy is improved via reinforcement learning beyond the demonstration quality. Thus, TiLoIL is neither a purely offline method nor standard behavioral cloning—it is an online, adversarial imitation learning approach specialized for structured, infinite-horizon LTL tasks. The primary contribution is to show that finite-length demonstrations can be used to overcome the exploration bottleneck inherent in learning LTL objectives over infinite horizons.

Q3. How are the product MDPs constructed, particularly in continuous domains?

W3. Use of LDBAs in continuous tasks: Was the state space discretized? Given that LDBAs are worst-case exponential in the size of the LTL formula, this may limit scalability to more complex objectives.

As is standard in RL for LTL tasks, we do not explicitly construct the full product MDP, which would be infeasible in continuous domains due to the infinite state space. Instead, we simulate product MDP transitions on-the-fly: at each environment step, we maintain a product state $(s, b) \in \mathcal{S} \times \mathcal{B}$, where $s$ is the current (continuous) environment state and $b$ is the current LDBA state. Upon taking an action, the environment transitions to a new state $s'$ ($s = s'$ if the action is an epsilon transition), and the automaton transitions to $b'$ according to the LDBA transition relation and the atomic propositions satisfied by $s'$ (Def. 2.3). This approach avoids constructing the full product MDP while preserving correctness, enabling us to learn policies over $\mathcal{S} \times \mathcal{B}$ that satisfy the LTL objective in a model-free manner. Crucially, we do not discretize the state space, as such discretization is computationally prohibitive and does not scale to high-dimensional environments.

Q4. What kinds of tasks is TiLoIL likely to succeed or struggle with, and why?

W4. Experimental setup and insights: While TiLoIL performs well on many tasks, the task selection could be more clearly motivated.

TiLoIL has been tested on all the benchmark environments used in prior work LCER [63] and DRL2 [6]. We exclude Minecraft and Pacman, as these are simple, low-dimensional discrete environments where exploration is not challenging. In fact, TiLoIL can solve these tasks even without demonstrations—random exploration is sufficient to encounter accepting states, enabling TiLoIL to learn a policy with performance comparable to the LCER baseline [63]. We do include the CheetahFlip environment from [6], while omitting the other Cheetah variant, which is structurally similar to DoggoNavigate in both dynamics and LTL structure. In the experiment section, our focus is on evaluating TiLoIL in tasks that involve nontrivial exploration challenges and diverse LTL objectives.

Overall, TiLoIL performs better when demonstrations are of higher quality. As noted in our response to Q1, TiLoIL's relatively weaker performance on CheetahFlip—compared to its ablated variant, though still outperforming all baselines—is primarily due to suboptimal demonstrations. These demonstrations can only achieve one or two rotations and fail to provide sufficient supervision for maintaining the momentum needed to consistently flip while standing on the back leg. Despite this limitation, TiLoIL surpasses the demonstrations, suggesting that its performance is bottlenecked by the quality of the demonstrations rather than its learning capability. In contrast, the ablated variant TiLoIL_noStage focuses on learning to repeatedly stand on the front legs, a posture from which maintaining or regaining forward rotational momentum is easier due to the task's dynamics.

TiLoIL is especially effective in tasks where reaching accepting states requires structured, multi-stage behavior, and where random exploration is insufficient. In such settings, even suboptimal demonstrations that partially reach accepting conditions can significantly help exploration. Conversely, TiLoIL may struggle in tasks where demonstrations fail to sufficiently cover the key transitions in the automaton (e.g., CheetahFlip), or when the LTL formula requires environmental events that demonstrations fail to expose. Overall, TiLoIL’s performance is influenced by both the quality of demonstrations and their coverage of the automaton's structure.

W2. Positioning within the literature

We will revise the paper to move the second paragraph of Appendix B into the main text and expand it to more clearly position TiLoIL in relation to existing approaches. For example, Temporal Logic Imitation integrates LTL-based task specifications with low-level control but assumes a pre-trained continuous controller, limiting its ability to improve low-level behavior. In contrast, TiLoIL jointly learns both high-level logic alignment and low-level control from scratch in an end-to-end manner. Similarly, Diffusion Meets Options, LTLDoG, and CTG are fully offline methods trained on large datasets (e.g., 6,686 trajectories in PushT), whereas TiLoIL operates in an interactive setting and learns from only a handful of expert trajectories (e.g., 5 in Carlo). This makes direct empirical comparisons challenging, but highlights TiLoIL’s efficiency in low-data regimes with structured supervision. We will also expand our discussion to more systematically categorize prior work by (1) their strategies to leverage LDBA structures to address the exploration challenge, (2) whether they leverage automaton structures for hierarchical control, and (3) whether they operate in an online or offline setting.