Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance

We are grateful for your thoughtful, detailed, and positive review. Thank you for the encouraging feedback. We are especially pleased that you found the paper well-written and a pleasure to read, and that you recognized our work as a well-motivated and reproducible contribution to the study of animal behavior.

Your concerns and suggestions are meaningful, and we agree that addressing them will improve the manuscript. We address each of your points below.

Overstatements: Thank you for this valuable feedback. You are right that some of our claims, such as "unprecedented resolution," are overstated. Our intention was to highlight the potential for large-scale analysis enabled by our method's efficiency, but we agree the language should be more measured and precise. We have revised the last sentence of the abstract as follows:

By eliminating annotation and per-frame optimization bottlenecks, Pose Splatter enables analysis of large-scale, longitudinal behavior needed to map genotype, neural activity, and behavior at high resolutions.

Additionally, we have revised our description of the visual embedding in the abstract:

We also propose a rotation-invariant visual embedding technique for encoding pose and appearance, designed to be a plug-in replacement for 3D keypoint data in downstream behavioral analyses.

Pose Estimation: We agree that our method does not perform "pose estimation" in the traditional, semantic sense of identifying and structuring body parts. It is more accurately described as implicit 3D reconstruction, from which we derive a holistic descriptor of pose and appearance. We use the term "pose" in a broader sense to refer to the animal's overall physical configuration. In our revision, we will clarify this distinction, explicitly stating that our method provides an implicit or holistic representation of pose.

Specificity to Animals: You are correct that our core pipeline is, in principle, general and could be applied to other non-rigid objects. However, it was originally designed for animal behavior because its key strengths directly solve the most significant challenges in that domain. Specifically, our method is template-free, making it immediately applicable to diverse species that lack standard 3D models like those used in human-centric research. Furthermore, it is annotation-free, bypassing the major bottleneck of defining and annotating animal keypoints. Therefore, while the algorithm is general, and could probably be used for other types of objects, its impact is probably most pronounced in the animal domain.

Limitations and Practical Use: Thank you for these important practical questions. We agree that a clearer discussion of the method's settings is crucial for readers, and we will add a dedicated section on these considerations in our revised appendix. Our goal was to design a system that is robust and flexible enough for real-world lab settings. For the number of views, our method is adaptable to various multi-camera setups, performing well with a sparse set of 4 to 6 views. The approach is also robust to the kinds of non-ideal lighting found in many lab settings and does not require specialized studio lighting. This robustness is largely thanks to our reliance on a powerful foundation model (SAM2) for segmentation; because such models perform well in varied lighting, the prerequisite of a reasonable foreground mask is achievable under a wide range of typical lab conditions. Finally, the camera arrangement follows the standard best practice for any high-fidelity multi-view 3D reconstruction task: placing cameras to provide comprehensive coverage and minimize self-occlusion.

Additional minor issues and questions:

• Three U-Net Modules (Line 129): The choice of three U-Net modules was determined empirically in early experiments. We observed that performance improved incrementally as we stacked more modules, probably due to the iterative refinement of the coarse visual hull, where each module improves upon the output of the previous one. We found that three modules offered a good trade-off between performance gains, memory consumption, and forward pass time.

• Training Loss Computation (Line 159): To clarify, the training loss is defined between the rendered output and a single, randomly selected input view for each training sample. We will add a sentence to the "Loss Terms" section to state this explicitly in the revised manuscript.

• Figure 2(b): Thank you for this suggestion. We will revise the caption of Figure 2 and the surrounding text to provide clearer context for the nearest-neighbor study upon its first appearance, so it does not feel out of place.

• Pretrained Encoder (Line 177): We apologize for this omission in the main text. As detailed in Appendix C, the encoder is a pre-trained ResNet-18. We will add this important detail to the main text for clarity.

• Vague Embedding Description (Lines 191-196): The section in the main text is a summary of the final step of our embedding process, with the complete technical details of the Spherical Harmonic expansion and adversarial PCA provided in Appendix C. We have revised the main text to offer a more intuitive explanation of the process, ensuring the core concepts are easier to grasp without needing to immediately refer to the appendix:

However, we found that these feature vectors still strongly encoded the azimuthal angle of the animal, possibly due to uneven lighting conditions across views. To remove this effect, we employ an adversarial formulation of principal components analysis (PCA) to find a 50-dimensional subspace of the feature vectors that contains a large portion of the variance of the input vectors and can predict only a small portion of the variance of the sine and cosine components of the azimuthal rotation angle. We take these 50-dimensional pose descriptors as the visual embedding. See Appendix C for more details.

• Contextualizing the 2.5GB VRAM Usage (Line 219): This is a great suggestion to better highlight our model's efficiency. As we state in the paper, our model is very lightweight, requiring only about 2.5GB of GPU memory (VRAM). For context, competing feed-forward models like PixelSplat and MVSplat consume at least 20GB of GPU memory, depending on the dataset and batch size, while per-scene optimization models such as Gaussian Object require over 10GB. We will add these numbers to the revised manuscript to make the practical advantages of our efficient architecture much clearer.

• Figure 5 (left panel): This panel shows the R2 (Coefficient of Determination) score for predicting the 3D locations of each individual animal keypoint using the visual embedding. Each dot represents a keypoint (left rear paw, beak tip, etc.), and its position on the y-axis indicates the percentage of that keypoint's positional variance in an egocentric coordinate system that the visual embedding can explain using a 5-nearest neighbor regressor. See Appendix C for details. We find that the visual embedding is sufficient to explain a large portion of the positional variance of the 3D keypoints.

• Lines 305–307: The example was chosen to illustrate a subtle difference (a head tilt to the left) that our method captured but the baseline missed. The position of the mouse in the arena is different between the Query, KP NN, and VE NN columns and so the nearest neighbor retrievals are visually distinct. However, note the leftward tilt of the head apparent in the bottom Query image. This tilt is absent in the top KP NN image, in which the mouse's head is pointed straight ahead, but visible in both VE NN images. This is the visual distinction we intended to highlight. We will improve the figure's annotations (e.g., with arrows and zoom-ins) to make the key difference more obvious.

• On the 5-Nearest Neighbor Regressor (Lines 321-322): Thank you for pointing this out. The process is detailed in Appendix C, but we had mistakenly dropped a reference to the appendix in the part of the main text that describes this figure. To clarify: for a given test sample's visual embedding, we find the 5 training samples with the most similar embeddings (via Euclidean distance). The final predicted 3D keypoints are then simply the average of the ground-truth 3D keypoints from those 5 neighbors. We will add this explanation to the main text.

• On the Reference Formatting: Thank you for catching this. We will expand all "et al." citations to include the complete author lists in the final version, adhering to the conference's formatting guidelines.

We sincerely thank the reviewer for the positive feedback on our paper's structure, motivation, and logical pipeline. We are glad you found the idea of using visual embeddings for behavior analysis to be reasonable and well-motivated.

We would like to address the weaknesses you raised, as we believe they may stem from a misunderstanding of our work's primary task, goals, and contributions.

Comparisons to Object Models (GRM, TRELLIS, HunYuan3D): Thank you for suggesting these powerful recent models. Unfortunately, the code for GRM has not been publicly released, making a direct comparison impossible. Regarding TRELLIS and HunYuan3D, while they are indeed powerful, they are designed for a fundamentally different task. These are generative models trained on massive datasets of 3D objects to create novel 3D objects from inputs like text or a single image. In contrast, our method is a reconstruction model designed for scientific applications that require the model to accurately recover the specific pose and appearance of an animal from a sparse set of views, where no ground truth 3D structure is available.

Despite this task mismatch, we took your suggestion and tested these models on our data. The results highlighted a critical issue: these models tend to hallucinate occluded parts rather than reconstruct them. For instance, they might invent a symmetrical backside for an animal when it's not visible in the input view. This leads to the generation of a plausible 3D animal, but not an accurate reconstruction of that animal's true size, shape, and posture in that specific moment. Because our goal is the precise 3D reconstruction needed for scientific behavioral analysis, this tendency to invent data makes these otherwise powerful generative models unsuitable for our specific application.

Comparison with GaussianObject: Regarding the comparison with GaussianObject (GO), while our method does not offer a dramatic improvement in final rendering quality, the motivation for our feed-forward approach was never about marginal gains in image metrics; it was about enabling scalability for large-scale scientific analysis. As we report in the paper, GO requires roughly one hour of optimization per scene, while our method's inference time is just 30 milliseconds—a speedup of over 100,000 times. For the behavioral neuroscience domain, where datasets often contain hundreds of thousands of frames, this difference is critical. Per-scene optimization is computationally prohibitive at this scale, making a fast, feed-forward model the only viable path for practical, large-scale analysis.

Training from Scratch vs. using a Pre-Trained Model: We appreciate the reviewer’s suggestion to use a pre-trained model, but a suitable “well-established base model” for our specific task does not yet exist. The models mentioned, such as TRELLIS and HunYuan3D, are designed for generating objects from single views or text and are not optimized for reconstruction from the kind of sparse, low-overlap multi-view data we use. In the absence of a pre-trained foundation model for our task, training from scratch is the only viable option.

Fortunately, our model is computationally lightweight, requiring only about 2.5GB of VRAM and a few hours of training time per dataset, making this approach highly accessible to researchers. Despite this efficiency, our model demonstrates strong cross-species generalization, as shown in Figure 3a and our quantitative results in Section 4.3. We therefore believe our approach is a practical and effective solution until large-scale foundation models for this specific task are developed.

Regarding the request to add more species beyond the three in the current work, we agree this would strengthen the paper and it is a goal for future work. As suggested by reviewer peFs, we are currently working to apply Pose Splatter to additional multiview videos with pigeons and monkeys. However, we wish to emphasize that publicly-available synchronized, multi-view data of freely behaving animals is relatively rare. We believe our current contribution of two new datasets that will be released publicly, is already a significant and impactful contribution to the field.

Video vs. Image Models This is an insightful question that gets to the heart of our contribution. Traditional action recognition models are fundamentally temporal: they process a sequence of frames to classify an action that unfolds over time. In contrast, our method is static: its primary function is to create a high-fidelity 3D representation of an animal's posture at a single, specific timestep. The advantage of our approach is the superior quality of this per-timestep representation, which serves as a better building block for any downstream temporal analysis. Additionally, the static model gives us the flexibility to perform filtering or smoothing as required by specific scientific questions.

Therefore, the unique contribution of our work is not to replace action recognition models, but to provide them with a richer, more informative input. By accurately describing how the animal is posed at every moment, we enable more powerful downstream models to determine what action is occurring over time.

Finetuned AnySplat

The initial results we presented were from testing the pre-trained AnySplat model directly, without any fine-tuning. We took this approach to directly answer your original question: whether leveraging the prior of a large, generalizable model would be more performant or efficient than training a lightweight, feed-forward model like Pose Splatter from scratch. From an efficiency standpoint, we believe a true "apples-to-apples" comparison necessitates using the pre-trained model as-is. The moment a large model requires fine-tuning, a process that for AnySplat demands over 40GB of VRAM and significant training time, it loses its primary advantage of efficiency over a model like Pose Splatter, which trains from scratch in a few hours with less than 3GB of VRAM. Additionally, the primary experimental results presented in the AnySplat paper are zero-shot evaluations. Therefore, our initial zero-shot experiment was designed to test the practical, out-of-the-box utility of these large models.

Nonetheless, we agree that it is a valuable experiment to see how much performance could be gained by fine-tuning AnySplat on our data. Following your suggestion, we proceeded with this experiment. We adhered to the recommended training protocols from the official AnySplat paper and codebase, stopping the fine tuning process once the validation loss plateaued and key validation metrics showed no further improvement.

The results are as follows:

Mouse

Finch

The results clearly show that while fine-tuning provides a marginal improvement, it does not dramatically boost AnySplat's performance on our dataset. We believe this is because AnySplat's architecture is fundamentally dependent on the geometric priors distilled from its pre-trained VGGT backbone. An examination of its loss function confirms a heavy reliance on these priors (e.g., for depth and camera parameters). As noted in the AnySplat paper (Table 4), the improvement in camera pose estimation over VGGT, a critical component for accurate 3D reconstruction, is limited.

Qualitative analysis of our fine-tuned results confirms this dependency. The characteristic artifacts of VGGT, such as the "onion-peel" effect where 3D surfaces from different views are inaccurately layered due to imprecise camera poses, remain largely unresolved after fine-tuning.

Therefore, we conclude that for challenging datasets like ours, which feature minimal overlap between views, simply applying or even fine-tuning pre-trained large models offers limited returns. These models remain heavily constrained by the priors of their backbones, preventing dramatic performance gains on out-of-distribution data. In such cases, training a lightweight, feed-forward model like Pose Splatter from scratch proves to be a more effective strategy for achieving high-fidelity results.

Furthermore, the significant computational cost of fine-tuning AnySplat (requiring >40GB of GPU memory and over 24 hours of training) compared to Pose Splatter (<3GB memory, a few hours of training) reinforces our claim that Pose Splatter is more computationally efficient for this task.

We thank the reviewer for the feedback. We are glad you found the direction of our work interesting and noted our strong performance on the animal datasets, including cross-category generalization.

We understand your concerns regarding technical contribution, experimental comparisons, and clarity. We believe these points may stem from a misunderstanding of our paper's core objective and the specifics of our evaluation. We would like to clarify these points below.

Standard Components: We respectfully disagree with the assessment that our work has limited technical novelty. While the individual components like shape carving and U-Nets are established, the core innovation of our work lies in their integration into a new, end-to-end framework that solves a problem existing methods do not: the annotation-free 3D reconstruction of diverse animals. Our primary contribution is not a new network layer, but a complete system that works "without prior knowledge of animal geometry, per-frame optimization, or manual annotations." This is a fundamental departure from methods that require species-specific templates or costly per-scene optimization. Additionally, the standard components in our work enable a fast, lightweight, high-performing solution that requires only 2.5 GB VRAM, compared to about 20 GB VRAM for PixelSplat and MVSplat. Furthermore, we introduce a novel pose quantification tool—the visual embedding—which we demonstrate outperforms traditional 3D keypoints in both human preference studies and downstream behavior prediction tasks. The other reviewers (peFs, r2qH) recognized the novelty and impact of both our overall system and the visual embedding, affirming that it is a significant contribution to the field of animal behavior analysis.

Choice of Comparison Baselines: We would like to clarify our choice of baseline comparisons, as this seems to be a point of misunderstanding. The suggested baselines, such as RoGSplat and Generalizable Human Gaussians, are explicitly designed for human reconstruction and rely heavily on human-specific priors like the SMPL body model, which are unavailable for our animal subjects. In fact, to even attempt a comparison, one must first check if it's possible to fit SMPL parameters to the animal data. We tried this, but the SMPL model failed to converge on our animal datasets, optimizing into distorted shapes that did not reflect the animals' true forms. Also, the central premise of Pose Splatter is its ability to work without any species-specific templates; therefore, comparing our general animal model to a human template-based model would be an inappropriate comparison. We chose PixelSplat and MVSplat precisely because they are generalizable, feed-forward models not tied to a specific category. While these models are generally used for reconstructing whole scenes, they are also known for reconstructing individual objects in the scene well and represent the current state-of-the-art for feed-forward, generalizable 3D Gaussian splatting.

Also, regarding the use of two views for these baselines, this was not an arbitrary or unfair limitation. We were following the original authors' recommendations, as stated in our experiment result section (Section 4.3.): "In line with each author's recommendations, we trained both PixelSplat and MVSplat with two input views." Furthermore, our experiments in Appendix G show that providing these models with more views did not improve their performance. These models are designed for high-overlap stereo pairs and struggle in sparse-view settings with minimal overlap like ours, regardless of the number of input views, as stated in Section 4.3.

We respectfully ask the reviewer to consider that our paper's primary goal is not to achieve a new state-of-the-art performance in general 3D reconstruction, but to propose a novel and practical system for animal pose quantification. The model comparison serves as a reference point, but our main contribution is twofold: first, achieving annotation-free 3D reconstruction (a task not addressed by previous animal pose models) and second, introducing the visual embedding as a new tool for behavioral analysis that outperforms sparse keypoint methods.

Unclear Description of Visual Embedding: As other reviewers did not raise this concern, we would be happy if you could point to any specific areas of confusion so we can address them directly in our revision. In the meantime, we hope a concise summary below helps clarify our method, which is also detailed across the main text and appendices.

The process involves rendering images of the 3D model from 32 virtual cameras on a sphere, passing them through a pre-trained ResNet-18 encoder to get 512-dimensional features, applying a Spherical Harmonic expansion and taking the squared magnitude of the coefficients to achieve rotation-invariance, and finally using adversarial PCA to produce the final 50-dimensional embedding. The experiments validating this embedding, including the human preference study and the behavior prediction task, are detailed in Section 4.3, with additional specifics in Appendices C and H.

Thank you for the detailed feedback and for highlighting the key strengths of our work. We appreciate your comment that we are tackling an important and challenging problem, and that our method offers an easier-to-adopt alternative to traditional keypoint or mesh approaches without requiring any annotations or species-specific information. Thank you also for noting our detailed explanation and the thorough validation across multiple datasets. We have addressed specific points in your review below.

Number of Cameras: You are right that our use of the phrase “a small set of calibrated cameras” in the introduction is vague. While we provide the specifics in the experiments section, we will clarify in the introduction of our final version that our method was tested using 4 to 6 cameras and that we achieve best results with 5 or more cameras, as seen in Figure 5b.

We chose a 6-camera configuration because it aligns with the standard established by Rat7M, which is one of the most widely-used benchmarks for this task. A 6-view arrangement is a common standard in modern neuroscience for achieving precise 3D kinematic tracking (e.g. Weinreb, C., Pearl, J. E., Lin, S., Osman, M. A. M., Zhang, L., Annapragada, S., ... & Datta, S. R. (2024). Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, 21(7), 1329-1339.). We will add a sentence to the paper to provide this important context.

Regarding two-camera stereo methods, our experiments with state-of-the-art models like PixelSplat and MVSplat highlight their core limitation for this task: they require substantial overlap between views to function effectively. This makes it difficult to reconstruct a complete 3D shape, as large portions of the animal's body are often occluded from any given pair of cameras. As our results show, their performance degrades significantly on our datasets where view overlap is sparse. Figure 12 further demonstrates that their performance does not improve even when given 3 or 4 views.

Finally, regarding the practicality for large-scale experiments, a 4-6 camera system does not present a major barrier in a laboratory setting. Once the cameras are calibrated and fixed, they can collect longitudinal data for long durations and across different subjects with minimal effort. In outdoor or wild contexts, however, setting up this number of calibrated cameras may be prohibitive. We will add the following text to describe these limitations:

Pose Splatter requires a minimum of roughly four calibrated cameras to operate, which may hinder its use in some outdoor or wild settings.

Multi-Animal Support: You've correctly identified that our current model is designed and benchmarked for single-animal scenarios. As we note in the discussion, handling the "prolonged and severe inter-animal occlusions" in multi-animal scenes is a significant challenge that we have not yet benchmarked. While we mentioned this briefly as an area for future work, we will state it more explicitly in a limitations section in our revised manuscript:

Additionally, the model has no mechanism to handle occlusions of the animal, which can occur in complex environments and multi-animal scenarios.

Metrics (L1, IoU, PSNR, $R^2$, etc.): We apologize for not defining the evaluation metrics more clearly. We omitted detailed explanations as they are standard in 3D reconstruction literature, but we agree that providing them would make the paper more accessible. We will add a new subsection to the revised manuscript defining each one.

For your convenience, we provide a brief overview of the benchmark metrics used in our evaluation tables. To compare performance against other methods, we use four standard metrics. The L1 metric measures the average absolute pixel-wise difference between our rendered image ($\hat{x}$) and the ground truth ($x$), evaluating color and texture accuracy, where a lower value is better. The IoU (Intersection-over-Union) score evaluates shape accuracy by calculating the overlap between the rendered silhouette ($\hat{m}$) and the ground-truth mask ($m$), where a higher score is better. We also report PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index measure), which are standard image quality metrics where higher scores indicate a better, more perceptually similar reconstruction. It is worth noting that the L1 and IoU metrics also form the basis for our color loss ($L_{color}$) and silhouette loss ($L_{IoU}$), respectively, which are minimized during training.

In our keypoint prediction experiments, we use the $R^2$ score, or the Coefficient of Determination. This metric measures the proportion of variance in the 3D keypoint coordinates that is predictable from our visual embedding. While the interpretation of what constitutes a ``good'' $R^2$ score is highly context-dependent, for complex and noisy biological data like animal behavior, a score above 0.5 generally indicates a strong predictive relationship. As shown in Figure 5 (left), our visual embedding proves to be a strong predictor for the animal's pose. For the finch, the $R^2$ values are consistently high (most > 0.75), indicating that our annotation-free embedding has captured the pose information almost as well as the manually derived keypoints themselves. For the mouse, we also see strong predictive power for most keypoints, demonstrating that our method successfully encodes quantifiable information about the animal's posture. This result is significant because it shows our embedding can serve as a rich, quantitative replacement for labor-intensive keypoint annotation in downstream analyses.

Other multi-view Datasets (3D-POP, 3D-MuPPET, 3D-SOCS): Thank you for suggesting these valuable datasets. It seems like the single pigeon videos from 3D-POP could be a valuable addition to the paper. We also agree that adding experiments using the OpenMonkeyStudio data would be useful. We have begun processing this data and hope to add updates during the Author/Reviewer discussion period.

3D Position Output: As stated in our methods section, the direct output of our network is a collection of 3D Gaussian particles representing the animal’s complete shape and appearance, which can be projected onto a 2D plane to create a rendered image from any given camera view. From a practical standpoint, the mean of each of these individual particles provides a real-world XYZ coordinate, and the full set of these points forms a representation of the animal's 3D shape that can be used to quantify movement.

Our focus in the paper is to quantify the pose of the animal without reference to animal location or orientation (azimuthal angle), both of which are estimated in a pre-processing step described in Appendix A. Pose Splatter uses these estimated values to produce predicted real-world 3D positions of the Gaussian particles that make up the animal.

Dataset Release and Documentation: You are correct that the mouse and finch datasets are new contributions of this work. We are fully committed to open science, and as stated in the paper, the datasets will be released upon publication. We will include videos, mask videos, camera parameters, estimated 3D center and azimuthal angles, visual embedding time series, example code, and documentation. We are planning to release the data on the DANDI archive in Neurodata Without Borders (NWB) format. The DANDI archive is data repository geared toward enabling reproducible science within the neuroscience community, where each dataset is assigned a DOI.

[fx 0  cx]
[0  fy cy]
[0  0  1 ]