Orient Anything V2: Unifying Orientation and Rotation Understanding

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: Overlooks several important related works

Thank you for your reminder. We will add discussions of these works and revise some of our statements in the updated version. While these works achieve a certain degree of open-world capability, their generalization to real-world scenarios remains limited by training scale and model capacity. We provide additional comparisons of median error (Med↓) below: (since NOPE did not release open-source checkpoints, we failed to reproduce its results during rebuttal)

W2: Source of improvement

Our improvements are demonstrated in three aspects:

1. Absolute orientation estimation (Table 1 and Sec 6.1),
2. Relative rotation estimation (Table 2 and Sec 3.2),
3. Rotation symmetry recognition (Table 3 and Sec 3.3).

The dataset mainly contributed to the absolute orientation estimation in Table 1. However, the improvements in relative rotation estimation and rotation symmetry recognition are due to the updated model architecture (supporting multi-frame input) and learning objectives (periodic angular distributions).

Therefore, across all three dimensions, both our new data engine and framework design are the main contributions.

W3 & Q1: Usage of VGGT pre-training

Firstly, regarding the baselines: Orient Anything v1 is initialized with DINOv2, while POPE uses pre-trained DINOv2 and SAM. Under a fair comparison setting with DINOv2 initialization, our method shows comprehensive advantages, as shown below:

Secondly, we believe that strong pre-training is the foundation for developing powerful models. For instance, VGGT itself is initialized with DINOv2, Depth Anything v2 heavily relies on DINOv2, and Segment Anything uses MAE and CLIP pre-training. We believe it is normal for performance to drop without pre-training, and it's valuable to demonstrate how far a model can go when built upon existing research.

Limitation 1: Does not estimate object translation

Object translation in images relates to 2D bounding box and depth, which correspond to object detection and depth estimation models, respectively. Given the strong generalization and modularity of these methods, our work focuses solely on object orientation estimation. In practical applications such as robotics and AR/VR, one can modularly combine object detection, depth estimation, and our orientation estimation model to obtain the object's absolute pose, including both 3D translation and rotation.

Thanks to the authors for addressing all the concerns raised by myself and the other reviewers. I’ve read through all the reviews and the rebuttal carefully. Overall, I found the authors well responded to the points raised, and I appreciate the addition of new experiments (e.g., initializing with DINOv2 and showing the effect of the proposed data engine, etc). Therefore, I’m inclined to raise my score.

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: Tasks depicted in the third column of Figure 1 are difficult to follow

Our intention was to use semi-transparent arrows to indicate the orientation of the previous frame and annotate the predicted relative rotation. However, due to lack of explanation and insufficient distinction between arrows, this may have caused confusion. In future versions, we will add clarifications and use more distinguishable arrow styles (e.g., lighter colors) to differentiate them.

W2: Confusing Lines 23-24

We intended to convey that while Orient Anything can predict absolute orientation, it does not fully capture relative rotation, despite the intrinsic link between object orientation and rotation. We will modify and polish these statements to eliminate confusion.

W3: Overly celebratory Line 46

Thank you for your suggestion. In future versions, we will use more neutral phrasing, such as "robust performance" and "precise results."

Q1: Extend to multiple objects

As discussed in Orient Anything V1, by equipping with object detection and segmentation models, we can isolate each object in multi-object scenes and perform orientation prediction separately. This simple pipeline yields strong results.

Q2: Application to arbitrary object classes not present in the training set

We tested on some imaginary objects (e.g., fantasy animals or vehicles) that do not exist in our training set. Our model is able to make reasonable predictions based on common sense, such as object parts or proportions.

Additionally, SynCLR [1], which also synthesizes text-image paired data from ImageNet21K class tags using LLMs and image generation models, demonstrates strong generalization to arbitrary categories. This may be due to the inherent diversity of image generation models. Our method contains similar category coverage as SynCLR, and thus can be expected to generalize similarly to arbitrary categories.

[1] Tian Y, Fan L, Chen K, et al. Learning vision from models rivals learning vision from data. (CVPR 2024)

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1 & Q1: Effect of Data Engine

Below are comparison results using the same number of object assets from different sources (our data engine vs. Objaverse):

The comparison shows that even without considering scalability, our data engine has significant advantages in asset quality and annotation accuracy.

W2 & Q2: Domain gap between synthetic and real data

We have adopted two strategies to improve domain transfer:

1. We mixed annotated real images (i.e., ImageNet3D) into our training data.
2. We initialized our models with strong base models pre-trained on large-scale real data, which has been shown to significantly improve synthetic-to-real transfer in works such as Orient Anything v1, Depth Anything v2[1], and Marigold[2].

Additionally, more domain gap mitigation methods can be explored. For example, we found that using advanced depth-conditioned image generation models can redraw rendered images into photorealistic ones while preserving object structure, which could be a promising direction for addressing domain gaps in the future.

[1] Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data. (CVPR 2024)

[2] Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation. (CVPR 2024)

W3 & Q3: World coordinate system

During rendering, we align the center of each 3D object's bounding box to the origin, so by definition, our orientation is aligned with the object's 3D box center.

However, our model only outputs three Euler angles and does not predict the 3D box center. For visualization, we currently overlay the orientation arrow at the center of the object's 2D bounding box for quick illustration. More precise alignment with the origin would require combining advanced depth prediction or monocular 3D box detection models to estimate the object's 3D box center.

Thank you for your supportive review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1 & W2 & Q1: More baselines in relative rotation estimation tasks

We report the median error (Med↓) of more baselines, Orient Anything v1, VGGT, and two versions of Relpose (co3dv1 and co3dv2), on the relative rotation prediction benchmark as follows:

W3: Lack of discussion on existing work on rotation symmetry and relative pose estimation

Thank you for your suggestion. We will revise our wording and add discussions of these papers in the updated version. In particular, Relpose [Ref1] demonstrates better generalization to open-world scenarios, and we further include it as one of the baselines for the relative pose estimation task. The comparison results are provided in our response to your W1 & W2 & Q1 above.

Q2: Open-sourcing the data engine and resulted dataset

We will do our best to open-source our work. Each part of our data engine is based on open-source models or methods, and we will organize and release our data engine pipeline. However, for the resulted datasets, the situation is more complex: our 3D object assets were generated using Hunyuan-3d-2.0, and we need to further confirm the license with the Hunyuan team before deciding whether to release large-scale generated 3D assets.

Q3: Only considering vertical rotation symmetries

To simplify the problem, we only considered vertical rotation symmetry, which covers most cases. However, as you pointed out, some objects may exhibit rotational symmetry along the left-right or front-back axes. We believe that multi-view rendering and prediction voting along these axes may help capture such symmetries.

Q4: How does the model perform for spheres

Thank you for raising this point. We tested several sphere cases and observed some interesting phenomena. For spheres placed on a plane (with background), the model predicts a pitch angle relative to the supporting plane (i.e., the object's up vector is perpendicular to the plane). For spheres without a background, the model does indeed predict a random 'up' direction, which makes sense because any angle is valid. This aligns with human perception and observation.