Agent-to-Sim: Learning Interactive Behavior from Casual Videos

Thanks for your constructive feedback! Below please find our responses to your questions and comments.

Q1 The paper focuses on environment-aware motion of agents in the presence of a (human) observer. Even if out of scope for this paper, it would be interesting to discuss more complex agent-environment interactions. How is an additional (moving) agent in the scene (e.g. a person or another animal moving in the background) currently handled by the 4D agent-observer-scene modeling method described for a single agent?

Our model is designed to handle interactions of a single agent with the observer and the scene. The other agents that appear in the videos are treated as part of the scene. However, due to the static scene assumption (Eq. 1), those moving agents are either averaged out or not well reconstructed (please see Fig. D in the rebuttal pdf for a visual example). Solving re-identification and multi-object tracking [E] in 3D will enable introducing multiple agents, which is an exciting future work.

[E] Rajasegaran, Jathushan, et al. "Tracking people by predicting 3d appearance, location and pose." CVPR 2022.

Q2 Are there many examples of videos not reconstructible due to notable changes in the scene layout (e.g. movement of large furniture) in the captured data? If so, how is it handled? I did not see any reference to this issue in the original manuscript, but I think it is a reasonable discussion when attempting to employ this framework at scale.

Thanks for the suggestion. As shown in Fig. D of the rebuttal pdf, our method fails to reconstruct notable layout changes when they are only observed in a few views, e.g., the cushion and the large boxes (left) and the box (right). Leveraging generative image prior to in-paint the missing regions is a promising direction to tackle this problem [F].

[F] Weber, Ethan, et al. "Nerfiller: Completing scenes via generative 3d inpainting." CVPR. 2024.

Q3 How could this framework be extended towards modeling complex interactions with the environment (e.g. opening a door, sitting on a chair after moving it, etc.)?

To reconstruct complex interactions with the environment, one idea is to extend the scene representation to be hierarchical (represented as a kinematic tree), such that it consists of articulated models of interactable objects [G, H, I]. To generate plausible interactions between the agent and the scene (e.g., opening a door), we can extend the agent state $G$ to include both the agent and the articulated objects (e.g., door), and learn a behavior generator to generate their trajectory jointly [J].

[G] Song, Chaoyue, et al. "REACTO: Reconstructing Articulated Objects from a Single Video." CVPR. 2024.

[H] Liu, Jiayi, Ali Mahdavi-Amiri, and Manolis Savva. "Paris: Part-level reconstruction and motion analysis for articulated objects." ICCV. 2023.

[I] Wei, Fangyin, et al. "Self-supervised neural articulated shape and appearance models." CVPR. 2022.

[J] Li, Jiaman, et al. "Controllable human-object interaction synthesis." ECCV 2024.

Q4 How many scenes and different agents are used in training and validation? Do specific agents overlap across splits, or just the types of agents?

We demonstrate our approach on four types of agents with different morphology, i.e., cat, human, dog, and bunny, in three different scenes, where human and cat share the same scene. We train an instance-specific model for each agent, and the data is not mixed up across agents. For quantitative evaluation of behavior generation, we report the performance of the cat agent on the held-out test sequences of the same cat. Training a model across different agent identities and types will be an interesting future work.

Q5 The current experiments use a small number of environments/scenes, which makes it hard to justify considering the system for larger-scale deployment.

For the cat dataset, we use 26 video clips over the span of a month, which is not super large-scale but we believe this is an important step to go beyond a single video. The major difficulty towards large-scale deployment is the efficiency and robustness of 4D reconstruction algorithms. In terms of robustness, we showed a meaningful step towards scaling up 4D reconstruction by neural initialization (Eq. 6). We believe this paper is a step toward large scale deployment, and we will release code for reproducibility and make it easy to follow up.

Q5 In the appendices (L.507) and implementation details (L. 245) the training time horizon is 6.4 seconds, but in L.181 the motion modeling sets a horizon of 5.6 seconds. Is my understanding correct that, for each window, the first 0.8 seconds are used as previous trajectory and the remainder 5.6 seconds as targets?

You are totally correct about this!

Thanks for your constructive feedback! We added a table of notations to improve the clarity, and we will expand the explanation of individual symbols in the paper.

Below please find our responses to your questions and comments.

Q1 What are the color and feature descriptors of the video?

Given a video $i$ with $N_i$ frames, we extract color images {$\{I_1, \ldots, I_{N_i}\}$} and compute their dense features descriptors {$\{\psi_1, \ldots, \psi_{N_i}\}$} using a pre-trained DINOv2 network.

Q2 What is X, and is it a point cloud obtained from a mobile phone? If so, how does the point cloud acquire its color in Equation 2?

${\bf X}$ in Eq. 1-3 are not point clouds. They are continuous 3D coordinates used to the define density and color of NeRFs in its implicit representation. The color at a location ${\bf X}$ can be queried by evaluating the MLP in Eq. 2. The parameters of the MLP are learned via differentiable rendering optimization in Eq. 7. We will clarify in the paper.

Q3 The paper lacks detailed quantitative experiments to demonstrate the effectiveness of the method.

We provided additional quantitative comparisons and analysis in the global response Table A-C, specially on camera localization, 4D reconstruction, and behavior learning. We found that our design choices are necessary to achieve good performance.

Q4 What is the practical use of this work?

Our goal is to learn “world models” that one can interact with from videos. This is a fundamental question that has practical application in generating contents for VR/AR, as well as robot learning with plausible agent simulation. For VR/AR applications, our approach enables generating data-driven agents that can interact with humans and scenes in a realistic manner. For robotics, the learned realistic behavior simulation can be used to pretrain robot policies that have a smaller sim-to-real gap, before adapting to the real world.

Thanks for your constructive feedback! We plan to expand on details in the additional page of the final version as well as the appendix. In the response to Reviewer C9Gr, we also added a table of notations to improve the clarity. Our code and data will be released for reproducibility and our goal is to allow researchers to continue working along this path.

Below please find our responses to your questions and comments.

Q1 Evaluation of the method seems limited and is mostly limited to qualitative comparisons.

We provided additional quantitative comparisons and analysis in the global response Table A-C, specially on camera localization, 4D reconstruction, and behavior learning. We found that our design choices are necessary to achieve good performance.

Q2 Why is each module in behavior generation evaluated separately, conditioned on GT inputs? Since the task is behavior prediction, another natural evaluation setting seems to be an end-to-end evaluation setting comparing body motion prediction from ego-perception inputs. This would open up other ablation studies to understand the efficacy of the hierarchical model; e.g., by comparing against a non-hierarchical diffusion model.

Thanks for the great suggestion! We re-did the evaluation of the behavior prediction using the suggested end-to-end setup without using GT goals/paths. As a result, hierarchical out-performs one-stage by a large margin for all metrics. We posit hierarchical model makes it easier to learn individual modules. Please see the global response for details.

Q3 What is the video code 𝛽 and how is it used?

The video code 𝛽 is a 128-dimensional latent code that is concatenated to the fourier code as the input to NeRFs, similar to GIRAFFE [B]. We use this idea to represent scenes with slightly different layouts given a shared NeRF backbone.

[B] Niemeyer, Michael, and Andreas Geiger. "Giraffe: Representing scenes as compositional generative neural feature fields." CVPR. 2021.

Q4 How are the ego-perception codes used in behavior generation?

The ego-perception codes are used as conditioning signals for behavior generation. Specifically, we concatenate the perception codes with the positional encoding of diffusion timesteps (representing the noise level $\sigma$) to predict the amount of noise. The predicted noise is then subtracted from the input noisy signal. This process is repeated for 50 times until a clean signal is obtained.

Q5 What is B in L182?

$B$ in $G \in \mathbb{R}^{6B \times T^*}$ is the number of bones of the agent. Each bone has 6 degrees-of-freedom including a center location (3DoF) and orientation (3DoF).

Q6 In L191-196, how are the two diffusion models used? Are they combined to use a form of classifier-free guidance?

Our behavior model consistents of 3 diffusion models, for goal, path, and full body motion generation respectively. Each diffusion model is trained with random dropout of the conditioning [C]. At test time, we classifier-free guidance to mix the conditional and unconditional score estimates with guidance-scale s = 2.5 following MDM [57].

To enable precise user control for the path and full body models, we follow controlnet [72] and omnicontrol [62] that use two networks with identical architectures and dense skip connections in between. The first network receives the perception codes $\omega$ only; the second network receives additional control inputs (i,e., goal and path) and modulates the intermediate features of the first network. We found the ControlUNet architecture allows precise control when goal and path is provided by the user, as shown in Tab. D in the response to Reviewer m6Ge.

[C] Ho, Jonathan, and Tim Salimans. "Classifier-free diffusion guidance." arXiv preprint arXiv:2207.12598 (2022).

Q7 FaF seems like a detection + multi-agent motion forecasting paper for self-driving, so it's not immediately clear how it can be adapted to this setting.

This baseline represents goal, path, and full body motion as Gaussians, and learns to predict both the mean and variance of Gaussian distributions by minimizing the negative log-likelihood [D]. We implemented and trained it using the same data as ATS.

This baseline was named as FaF because its input/output are close to FaF’s motion forecasting module. To avoid confusion, in the new Table A and Table D, we renamed it as Gaussians.

[D] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." NeurIPS 2017.

Thanks for your constructive feedback! Due to the complex nature of the problem, it is difficult to unpack all the details in the limited space. We plan to expand on details in the additional page of the final version as well as the appendix. In the response to Reviewer C9Gr, we also added a table of notations to improve the clarity. Our code and data will be released for reproducibility and our goal is to allow researchers to continue working along this path.

Below please find our responses to your questions and comments.

Q1 How are the agents, and especially the animals, modeled? A particular aspect which is not addressed in detail is the modeling of the agents, especially of animals like cats that are quite challenging due to their non-rigid nature.

We use the bag-of-bones model from BANMo [65], which accounts for both articulated motion and non-rigid deformation. The deformation is computed by blending the motion of a set of unstructured 3D coordinates/bones that rotate and translate over time.

Q2 In particular, it is not clear how eq.2 is combined with eq.3, and why the same number of "bones" (b=25, L.137) is used for all agents. Also, the nature of G^b is not discussed in detail.

We model the density and color of an agent in the time-invariant space (Eq. 2), which can be mapped to the deformed space at a given time instance (Eq. 3). We use $B=25$, as it is the superset for bones of all agents we processed. $G^b_t$ is a rigid transformation representing the state of bone b at time t.

Q3 How NeRF-type reconstructions are combined with feature descriptors, and how this helps in handling layout changes?

Similar to distilled feature fields [A], we extend a static NeRF to represent feature fields (Eq. 1), and optimize them together with the motion $\mathcal{D}$ using DINOv2 descriptors. This is referred to as feature-metric bundle adjustment (FBA) in Eq. 7. We find FBA is robust to moderate layout changes, since DINOv2 feature descriptors are robust to local appearance changes [40]. We quantitatively validate the effect of FBA in global response Tab B. This is also evident in Fig. D of the rebuttal pdf, where we can localize cameras despite of the layout changes of the scene.

[A] Kobayashi, Sosuke, Eiichi Matsumoto, and Vincent Sitzmann. "Decomposing nerf for editing via feature field distillation." NeurIPS 2022.

Q4 Clarification on designs (Eq. 6-7, Eq. 10-11).

We added an ablation of Eq. 10-11 in Table D row (d). We find that replacing ControlUNet with concatenation (L197-198, concatenating goals with perception codes) produces worse results (e.g., Path error: 0.115 vs 0.146). We also provided additional quantitative comparisons and analysis on camera localization (Eq. 6), feature-metric bundle adjustment (Eq. 7) in the global response Table B. We found that those designs are necessary to achieve good performance.

Table D: Evaluation of Behavior Control. We separately evaluate path and full body motion generation, given guidance signals of goal and path respectively. The metrics are minimum average displacement error (minADE) with standard deviations (±σ). The best results are in bold.

Q5 What happens if a behavior is observed only once in the dataset or conversely, how many times does a behavior need to be observed to be included in the model?

Due to our ego-centric encoding (Eq.12), we find that a behavior can be learned and generalized to novel situations even when seen once. Although there's only one data point where the cat jumps off the dining table, our method can generate diverse motion of cat jumping off the table while landing at different locations (to the left, middle, and right of the table). Please see Fig B of the rebuttal pdf for the corresponding visual.

Q6 How robust is the method with respect to changes in the environment?

We noticed displacements of chairs and the presence of new furniture in our captured data. Our method is robust to these in terms of camera localization (Tab B of the global response, Fig D of the rebuttal pdf). However, 3D reconstruction of these transient objects is challenging and we leave it as future work.

Q7 What "longitudinal videos" are?

Longitudinal videos come from the term “longitudinal study”, which refers to a research design that involves repeated observations of the same variables (e.g., people) over long periods of time. We think it fits well with our study of learning a behavior model of agents from videos captured over a long-horizon. We will clarify.

Q8 What does "low latency" mean in this context?

Low-latency indicates the model can generate goals at an interactive frame-rate. We will clarify.

I thank the authors for their responses and their comments. I appreciate the additional results provided as well as the evaluation regarding camera localization and 4D reconstruction. Their answers have clarified some aspects, especially regarding agent reconstruction. As mentioned in the reviews, there are issues regarding notation and clarity which, to some extent, are due to the limited space available for describing in sufficient details all the claimed contributions. In my view, the one additional page of the final version will still not be sufficient. This also concerns reproducibility, as sufficient details in the paper itself need to be provided, even if the code will be published. I increase my rating to borderline reject, as I appreciate the contributions of this work and the answers provided by the authors, yet I feel that a more detailed presentation and evaluation would make the paper much stronger.