Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos

Thanks for the encouraging comments and constructive feedback. We have revised the paper to address the confusions about neural camera localization, notations, and method details. Key text changes are highlighted in dark red in the revision.

Below, we provide answers to the questions raised and would welcome any further feedback or clarification requests.

Q1.

It is not clear how to collect the training data for neural camera localization. Does the camera move along with the agent or fixed during capturing？ If it is fixed, there are only a few camera poses that can be used in the training. If it can move, that means we already have an algorithm to obtain accurate camera poses for training.

The paired GT poses and images are synthetic data generated from a 3D map of the room. The 3D map is built using Polycam from a single video, without the agent. During training, we generate random camera viewpoints and render images on the fly given the 3D map, and the localizer learns to regress the camera rotation and translation from a DINO image. Then, neural localizers are used to initialize the optimization in Eq. 6 across all the videos. This mechanism enables us to obtain good initialization for the target videos to register them in the consistent world coordinate. We revised the paper in L224-237 accordingly, and added Fig. 8 and Fig.9 in the appendix to illustrate the process of training neural localizers and using them to initialize optimization on target videos.

Q2.

The symbol \sigma is used to represent density in Sec. 3.1 and noise in Sec. 3.3, a kind of misleading when reading the paper.

Thanks for pointing it out. We have replaced $\sigma$ with $\rho$ to denote density in the revision to reduce confusion.

Q3.

In eq.8, both \sigma and \epsilon indicate the noise? since it is mentioned in line 303 that \sigma is a noise value. If it is the case, please clarify: Does \epsilon indicate the variance of the noise?

$\sigma$ is the standard deviation of the Gaussian noise (dependent on the timestep in the diffusion process) and $\epsilon$ is the actual noise that is sampled from the gaussian distribution. We have clarified in L306 of the revision.

Q4.

In line 310, how large is the volume queried from the 3D feature volume?

The local feature volume is queried with a grid ${\bf X}^a \in \mathbb{R}^{64\times8\times64}$, which encodes a $6.4\mathrm{m} \times 0.8\mathrm{m} \times 6.4\mathrm{m}$ box around the agent along the width (X), height (Y), and length (Z) dimension. This has been included in L854-855 of the revision.

Q5.

What is the definition of the generated path? Is it the root translations in the scene？

This is correct and it has been clarified in L261-264 of the revision.

We hope these revisions and explanations address the concerns raised. Please let us know if there are further questions or aspects needing clarification.

Thanks for your constructive feedback. Based on the comments, we have revised the paper to address the noted weaknesses in writing. Key text changes are highlighted in dark red in the revision. Below, we provide detailed answers to the questions raised and would welcome any further feedback or clarification requests.

Below, we answer the questions about technical details.

Q1.

“Neural localizer is found to be more robust than geometric correspondence while being computationally efficient” is not well grounded through technical principles or empirical evidence.

Geometric correspondence methods, such as DUSt3R (Wang et al., 2023), scale with $\mathcal{O}(N^2)$ memory and computation for $N$ images, which becomes infeasible for large-scale datasets (e.g., 10k images). In contrast, registering each image to a canonical representation reduces the cost to $\mathcal{O}(N)$, making it significantly more efficient and feasible to run at scale. We clarified this point in L221–224.

Q2.

The ego perception of the world and scene, observer, and past encoding needs more elaboration in writing to make them understand. It is not clear how the path is generated given a goal? Is this the shortest path or any other path?

The path represents the root joint locations generated by our diffusion model. At test time, we apply DDPM sampling: initializing with Gaussian noise in trajectory space and iteratively denoising it using the learned score functions (Eq. 9). The training of the path and body diffusion models follows the same procedure as the goal diffusion model. We expanded the description in L290–302.

Q3.

I assume Equation 1 and 2 are through NERF context. it is not clear to train them, how to get the poses to train them?

Our 4D representation (scene NeRFs and agent NeRFs, and deformation fields) is optimized using Eq. 6. The scene camera poses are initialized from the neural localizers and the agent camera poses are initialized from the pose regressor. The optimization leverages differentiable rendering to optimize NeRFs and deformation fields against video observations, including color, flow, and depth. We revised L195-201 and added further details on compositional rendering in Appendix A.1. We also added Fig. 8 in the appendix to better illustrate the process to obtain and use neural localizers to initialize optimization on target videos.

Q4.

How to train Equation 2 when agent is in motion. Elaboration is needed. A picture is needed to show the representation of agent. It is not clear what is meant by root and how that is identified from video automatically for any morphology.

We use BANMo to learn the deformation model of the agent, as clarified in L181–194. BANMo jointly optimizes bone articulations and skinning parameters, where the "root" is the canonical agent coordinate, typically the agent's center. An illustration figure of the agent representation is included in the Appendix Fig. 7.

Q5.

Saito et. al. uses SMPL morphology where getting the skinning is understood. Here the agent can be anything. So the skinning part needs more elaboration to understand by the reader. What is meant by "bones have time varying centers" and how this impact the skinning? Need picture with explanation. What do we mean by set of 3D Gaussian as bones? How they are fitted in video? How they are initialized?

We use BANMo's skinning model. Given the bone locations and scales, skinning weights are computed as the Mahalanobis distances between a point and bones, normalized by Softmax. The bones are initialized uniformly on a sphere and optimized with inverse rendering. We revised the paper in L181–194 and a visual illustration can be found in Appendix Fig. 7.

.

Thanks for the clarifications.

If we know the M videos are all corresponds to the 3D map generated using Polycam then why those scenes are again reconstructed? If these videos can be different than the regions covered by the polycam, then how the new poses for the videos are obtained?

The relation between the GT 3D entities, videos, generalisation and assumptions needs to be bit clear for the pipeline to be applied in the novel set of videos where we dont have the GT polycam, and only have the videos over time.

Secondly, the re-encoding of the scene for egocentric localisation can be made a bit simple. Encoding the same scene in two different way may lead to computationally inefficient. Is there way the explicit scene geometry can be leveraged rather than implicit?

Also, SfM/SLAM methods can create the 3D in different coordinate for different videos. But if they are of same scene, they can be registered through rigid transformation. What will be the difficulty in this w.r.t line 208-209

Now few things are getting cleared.

However, if you use the 3D map generated using Polycam as an initial structure and use that to modify based on scene structure then the 3D change along with previous static 3D structure registration has to be detailed in Canonocal structure section. Right now equation 1 and 2 says that the entire structure is trained using MLP from video. The initial 3D is not reflected there and also how to change that 3D based on M videos is also not detailed. This is a very important step and initial 3D assumption in the entire methodology. This will help to understand Q12 above also in a more detailed way..

If you have the 3D scene with the changes, it will be good to comment why resection methods wont work for pose estimation after making the 3D scene complete with the new changes. This will scientifically ground the equation 5.

Thanks for the additional comments and we hope the following response addresses them.

Q14.

However, if you use the 3D map generated using Polycam as an initial structure and use that to modify based on scene structure then the 3D change along with previous static 3D structure registration has to be detailed in Canonocal structure section. Right now equation 1 and 2 says that the entire structure is trained using MLP from video. The initial 3D is not reflected there and also how to change that 3D based on M videos is also not detailed. This is a very important step and initial 3D assumption in the entire methodology. This will help to understand Q12 above also in a more detailed way..

To clarify, the canonical structure is not initialized from the template 3D map. To train the canonical structure, we follow a typical NeRF training setting, where the MLPs are initialized with random weights, and optimized by fitting to the image observations using differentiable rendering. In our case, the MLP weights are optimized together with camera poses (fine-alignment), which are initialized from the neural pose regressor (coarse-alignment).

To avoid confusion, we have clarified the initialization of MLPs in L181 and strengthened the connection to NeRF in L172 and L197-198.

Q15.

If you have the 3D scene with the changes, it will be good to comment why resection methods wont work for pose estimation after making the 3D scene complete with the new changes. This will scientifically ground the equation 5.

Combining the changes in the new videos and the template 3D map needs registration. However, registration is challenging due to non-stationary points and correspondence, as discussed in the previous response. Our method solves registration problem by training a scene-specific neural localizer that bypasses correspondence and rigidity assumption, and provides a good camera initialization for fine-registration (via inverse rendering).

We have updated L206-214 to make the motivation clearer.

We are happy to answer any further question or aspect that needs clarification.

We appreciate the discussion on how to leverage the template.

Q16

I understand the response from Q14. But that is also my precise doubt. Why template 3D map cannot be used when it is available? and use of M videos to incorporate the changes. This will be helpful for the reader to understand the motivation of not using that when available. Isnt make easy for the method to get the representation? A detailed discussion may help

Using a template 3D map for initialization is a good idea when there is little changes in the scene. When there are noticeable scene changes, starting from a template has the risk of overfitting, where it is more difficult to revert the changes and fit the new content (as the MLP has converged), compared to starting from scratch.

Please let us know if you have additional comments.

Thanks for raising the score. We try to address the remaining concerns.

To summarize the discussion, the reviewer's question is "Does having a 3D scene template good enough for registration and reconstruction of 3D structures from multiple unposed videos?" We list our understanding of the alternative pipeline being suggested below,

1. Pose the template and target videos using off-the-shelf tools (e.g., Polycam), and extract 3D structures.
2. Registration. Register the target 3D structures to the template 3D structure.
3. Reconstruction. Use the template 3D structure to initialize NeRF and train it on all videos.

In this pipeline, step 2 (registration) is challenging for the following reasons:

1. Geometric registration methods, such as ICP, need good initialization. Since the start location of video recordings are different, the 3D structures are initially largely misaligned, which causes ICP to fail.
2. Feature-based registration methods, such as HLoc, requires good visual correspondence, and finding visual correspondence is hard due to two reasons.
   • (1) Lack of view overlap between videos, e.g., the template video captures the front view of the sofa, but the target videos contain the side view of the sofa.
   • (2) Changes in appearance across videos, (e.g., lighting, table cloth gets replaced) and layout (e.g., furniture get rearranged).

We experimented with ICP and HLoc and both produced gross errors. Our method trains a neural localizer to provide initialization for registration, which leads to good registration and reconstruction results (Tab. 1, Tab. 2). We are happy to try and compare if the reviewer has alternative methods in mind for step 2.

To answer Q4, which is related to step 3 (reconstruction),

4.Nerf needed the camera pose and that anyway needs to be given so why not start with Polycam output.

Although we don't directly use the 3D template to initialize the scene structure, the posed template video is used for reconstruction. We use both the template video and new videos to train the scene NeRF. The camera pose of the template video is set to ground-truth cameras from Polycam, and the camera poses of the new videos are initialized from the neural pose regressor and optimized jointly with the scene NeRF. This leverages the accurate pose from Polycam, and avoids overfitting to the template scene.

We have revised the manuscript L207-214 and L250-253 to highlight this.

Thanks for the encouraging comments and detailed feedback. We have revised the paper to address the comments and provided additional details for reproducibility. Key text changes are highlighted in dark red in the revision.

Below, we provide answers to the questions raised and would welcome any further feedback or clarification requests.

Q1

How are the NeRF representations of scene and agent combined in detail?

We revised L193-199 and provided additional details on compositional rendering in Appendix A.1.

Q2

Reference of figures in the supplement.

We have clarified that the figures are part of the appendix in the revision.

Q3

The orientation representation for regression in Eq. 5.

We use unit quaternion to represent rotations. We force the real part to be positive to avoid the ambiguity in the representation, as quaternions q and -q represent the same rotation. This has been clarified in L228-229 of the revision.

Q4

It is not clear why there is a need for a joint optimization of poses and representations T and D as indicated in Eq 6, because the video sequence is annotated with camera poses from a different (unspecified) system (l. 225). Please clarify which other method is used to obtain the camera motion trajectory. Why is it necessary to optimize the poses in Eq. 6 ? How would the approach perform without this post-optimization? Please add an ablation study. What is the point of training a neural localizer if the camera poses are known from a different system?

Although one could obtain camera trajectory in a single video using off-the-shelf SfM/SLAM tools, when optimizing multiple videos of the same scene, the per-video trajectories are not globally aligned to the same coordinate system. The joint optimization aligns camera poses from all videos to a global scene coordinate system. Without Eq. 6, our method degrades to Multivideo-TotalRecon and produces worse reconstruction results (DepthAcc: 0.708 vs 0.099), as shown in Tab. 3. We have clarified the motivation of multi-video joint optimization in L202-210 of the revision.

Q5

How are the agent root and body part poses initialized for Eq. 6?

The body part poses are initialized as identity and the root pose is initialized from pose estimator. We have clarified in L245-246 of the revision.

Q6

How are agent and static scenes segmented?

We use Grounding DINO to annotate a bounding box given text description of the agent (e.g., cat), and SAM to segment the agent in the first frame of the video. The segmentation is tracked over all the frames using XMem. We took the existing code from Lab4d, which is an open-source project. This has been clarified in L845-849 of the revision.

Q7

Is the training and swapping of beta performed across various scenes (i.e. different types of rooms) or only within specific scenes?

The training and swapping is performed for the same scene across multiple videos and days.

Q8

l. 263, why was T*=5.6s chosen and how does the performance of the method depend on the horizon length? This should be evaluated as supplemental results.

We evaluated the behavior prediction with different prediction horizons T*={4.0, 5.6, 7.2}s, and results are updated to Tab. 4 and L482-483 of the revision. We found the longer the prediction horizon is, the more difficult to predict the goals and future paths. T*=5.6s was chosen following prior works (Rempe et al. 2023), which strikes a balance between the generation time and the coherence of the generated content.

Q9

Fig. 2, it is not clear what the thin black arrows mean (e.g. between the motion fields and between score map and goal/path). What is the motion field depicting? Where is this output in the proposed method?

Black arrows in Fig. 2. visualize the output of the denoising network (Eq. 7), which points to the direction to update the goal locations in the iterative denoising process. This has been clarified in L284-285 of the revision.

Q10

l. 304ff., the note on generalization to different actions seems problematic. The datasets used in the experiments seem small. The biggest dataset (cat) has only 22 training videos of a few minutes each. The anecdotal result pointed to in the supplemental material (Fig. 11) might be a rare example. Please discuss and revise the claims.

Thanks for pointing it out. We acknowledge that the datasets used in our experiments are relatively small, and we understand your concern that the anecdotal result in Figure 11 may not provide sufficient evidence for broad generalization. We have revised the claim about generalization from our paper. Instead of stating “We find that a specific behavior can be learned and generalized to novel situations even when seen once”, we revise it to “This enables the model to generate novel scenarios that were not present in the training dataset.” in L308-309.

Q11

Eq. 11, what is \Psi_s ? Please define.

$\Psi_s$ is the scene feature queried at point $X_w$ through= Eq. 1. We have clarified it in the revision L314-315.

Q12

l. 314ff, the text mentions trajectories, but the equation only transforms a single pose (\xi^a). Please clarify.

Thanks for pointing it out. This has been revised in L320-353.

Q13

l. 318ff, please explain why the approach uses the observer trajectory perceived by the agent and not the other way round. Please also provide an evaluation if this makes a difference.

The alternative way of encoding the state of the agent with respect to the observer is equivalent to our formulation when there is only one “observer/external agent”. However, this alternative formation is not extendible to more than one observer (external agent), while our formulation could be more intuitive when handling more than one external agents.

Q14

l. 360, a segmentation method is mentioned,but it is not stated how it is used.

We use the segmentation as the ground-truth for object silhouette loss in L249, which is necessary to separate the agent from the background. We updated the text from "silhouette loss" to "segmentation loss" to make it clearer.

Q15

l. 401. GT relative camera pose is obtained by some unspecified process. How does it work in detail and how accurate is this process? Please provide this description in the supplemental material.

We have updated the annotation process of GT camera pose in the appendix L863-902. The accuracy of camera pose annotation depends on the precision of 2D keypoint annotation and accuracy of the depth map. Solving relative camera poses has an exact solution when both measurements are perfect an d the point pairs are not degenerate.

Q16

l. 428ff / Tables 4 and 5: the motion prediction is conditioned on ground-truth goal and path for the evaluation. However, this should also be evaluated for regressed goal and path to fully evaluate the proposed method without such oracle information. How were ground-truth goal and path annotated in the data ?

Thanks for catching this. We made a mistake in writing and we’ve updated the text. In Tab. 4 "End-to-end Evaluation of Interactive Behavior Prediction", the predicted behavior is not conditioned on oracle information (GT goal or path). In Table 5 "Spatial Control", the path generation is conditioned on GT goal, and the full body motion generation is conditioned on GT path. The GT goal is the position of the agent’s root T*=5.6s into the future, and GT path is defined as the trajectory of the agent’s root T* seconds into the future.

Q17

It seems the paper lists several datasets (bunny, human, etc), but only shows results for the cat videos.

We show visual results of cat, bunny, human, and dog on the supplementary webpage. It includes 4D reconstruction for all video sequences of all agents, and behavior generation results for cat and human.

Q18

If possible, the datasets used in the evaluation should be made publicly available for future comparison with the proposed method. Is it planned to release the datasets publicly ? Please also specify how this data was collected. Are there potential personal data protection issues for releasing the data?

The data is collected from the friends of the researchers that contributes to the project, and the dataset will be made publicly available upon paper acceptance.

Q19

The datasets used for evaluation are rather small and, hence, results seem anecdotal and significance is unclear. The method should be evaluated quantitatively on more and longer sequences and a larger variety of agents (e.g., 10-20 different animals and human subjects) and scenes (e.g. 10-20 different rooms) to increase significance of results and to provide further insights into the performance and limitations of the proposed method. Also the variety of daily activities should be increased and annotated. Cultural and gender diversity of subjects should also be considered when designing a benchmark dataset. With larger datasets, also the dependence of the performance of the method on the amount of available training data could be evaluated.

We appreciate the valuable feedback regarding the scope and scale of the datasets used for evaluation. While we acknowledge that larger and more diverse datasets could provide additional insights and strengthen the evaluation, we would like to clarify the focus and constraints of our work. Our goal was to study this understudied problem of learning behaviors of a single individual from long observations. While there are studies that collects large-scale behavior datasets, learning from a videos of the same agent is less studied and we are in this camp.

We agree that creating a comprehensive benchmark dataset with diverse subjects, activities, and environments would be valuable. However, constructing such a dataset at the scale suggested is nontrivial, particularly around privacy concerns. While this is outside the immediate scope of this work, we believe that open-sourcing our code will enable the community to build upon our framework and contribute to the development of larger, more diverse benchmarks collaboratively. We hope this clarifies the intent and scope of our work.

Minor comments

Thanks for the feedback! Those have been fixed in the revision.

We hope these revisions and explanations address the concerns raised. Please let us know if there are further questions or aspects needing clarification.

Thanks for the insightful response which addresses several of my concerns well.

Further comments/questions:

• Results which are only provided on a volatile supplemental webpage cannot be used in a published paper to support claims, since the webpage is not provided with the paper by the publisher. Hence, please restrict the claims/result discussion to the datasets used in the main paper (cat; bunny for the specific experiment in Fig. 4).

• How is the ground truth for the agent motion obtained/annotated in the dataset? Please detail in the paper or supplemental material.

Thanks for the encouraging comments! We have revised the paper to address the feedback. Key text changes are highlighted in dark red in the revision.

Below, we provide answers to the questions raised and would welcome any further feedback or clarification requests.

Q1

Given that the agent in the demo appears somewhat unrealistic, I was curious why parametric models weren't used for agent representation. For example, there are parametric models available for animals, such as [1,2].

While parametric body models have their advantages, they also limit the type of animals that can be represented in nature. For example, bunnies and mice are outside SMAL family. To demonstrate that our approach is applicable to a wider range of animals, we use BANMo (NeRF+articulated bones) since it is flexible to explain more animal categories. We have also revised the paper and discussed [1,2] in the related works L113-118.

Q2

Regarding the egocentric perception: is the egocentric visual/video data encoded as well? It is mentioned "a latent representation from a local feature volume around the agent", but an agent like a cat can not see the scene behind it.

We do not record/use egocentric RGB videos. The iPhone (third-person view) videos are lifted to 3D (Eq. 1-2) and encoded as the scene input to the behavior prediction model. The latent representation of the scene in Eq. (11) is a 3D convolution of the DINO-v2 feature volume of the room in the agent's perspective. This encoding mostly captures the immediate surroundings of the agent. Encoding the visible surroundings might be a better design choice, but it is also not perfect due to the lack of other sensor modalities of the agents, such as audio and touch. We leave this for future investigation.

Q3

The proposed egocentric encoding to transform the world to the egocentric coordinates to avoid overfitting as well as past encoding to capture the past motion sequence seem to have been proposed and well discussed in [3], which also predicts the motions from spatial controls.

Please note that our model predicts future behavior of an agent from synthesized egocentric perception, as opposed to works like [3] which predict the full body motion from head-mounted sensor inputs. [3] is also a concurrent work that appears in ECCV 24 (which is after the ICLR submission). We appreciate the pointer and have cited it in the revision L307-308.

Q4

The proposed system is trained and tested on the collected dataset. However, based on the information provided in the paper, this dataset appears quite small and lacks diversity, as it includes only 4 agents, 3 scenes, and has a limited duration. Given the availability of numerous large datasets featuring moving agents and varied environments, incorporating these existing datasets for evaluation would strengthen the findings. Additionally, I am curious about the generalization capability of the proposed method, as the experiments were conducted by training on only 22 videos and testing on a single remaining video. Do the training and testing data contain the same scenes? If so, this might lead to potential overfitting issues.

We appreciate the valuable feedback regarding the scale of the datasets used for evaluation. While we acknowledge that using a larger and more diverse datasets could provide additional insights and strengthen the evaluation, we would like to clarify the focus of our work. We are exploring a novel problem setup, where video observations of the same agent across a long time horizon (weeks) is available. The goal of the model is to overfit to a single agent and scene. Therefore, we only study generalization to different time instances. To show the generality of the algorithm, we experiment our approach on a wide variety of agents: cat, human, dog, bunny.

Q5

I am interested in the potential applications of the proposed systems beyond the general areas mentioned in the paper, such as driving, gaming, and movies. It would be valuable to explore specific existing tasks to demonstrate how the system can provide practical benefits.

Thanks for the comment. We also hope the methods proposed in this paper will be useful for generating plausible behaviors for characters in gaming and interactive movies, as well as for embodied agents in the physical world. We leave this for future investigation.

We hope these revisions and explanations address the concerns raised. Please let us know if there are further questions or aspects needing clarification.