Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

Thank you for your positive review! We have updated our draft to incorporate your feedback.

When does the retrieval process fail?

As Fig. 7 shows, the retrieved objects indeed might not fully resemble the real-world objects (for example see the chair). This is quantified by the Chamfer distance in Table 2 (Appendix A.5). Nonetheless, we note that our articulation system is general purpose and can be integrated with any mesh acquisition method. Specifically, we have added new results in Fig. 6, 22, and our website where meshes generated by a diffusion model fully resemble the real objects.

Robotic experiment details

The simulation assets are generated by Articulate-Anything given in-the-wild videos as shown in the updated Fig. 13. We have spent significant effort developing and testing the control stack for our hardware system and have now included additional results showing direct sim2real transfer. Please see the updated Sec. 6 and website. We further show that compared to human labeling, Articulate-Anything saves, on average, 40.58 minutes of annotation time per task.

Other questions

• Clarification on 3D positioning

The link placement works by aligning the child and parent's link centers along an axis and performing collision checks to ensure tight placement without intersection. Thank you for your insightful observation! For the specific Fig. 4(a) case, this is due to the specific way that the PartNet-Mobility mesh is off-setted in their dataset so not all meshes are centered at the origin. In general, more granular positioning such as placement=left_front can be implemented in the API. Alternatively, our VLM can be prompted to chain front and left placements together to achieve the same effect using in-context learning.

• Number of actor candidates

The actor generates python code until one code candidate can be run without any compiler error. The maximum number of candidates is set to 3. We have found that our system can generate at least a compilable program on the first try in most cases, or on a second try in the worst case.

• Can the API support all cases?

Both the link placement and joint prediction APIs can cover all possible cases without loss of generality. For example, for link placement, we have an optional clearance (i.e. offset) parameter that can be added to the placement. Placements can further be chained together by making multiple python calls, thereby allowing for a link to be placed in any position in the 3D space.

Thank you for your constructive feedback! Here we addressed your comments:

Articulated objects in the evaluation

We use the PartNet-Mobility dataset, which is the largest human-annotated and most widely used dataset for papers in this topic (see URDFormer, Real2code, PARIS, Ditto ect.). This dataset includes human annotations for everyday objects, which include the most common joint types: prismatic and revolute. We have not evaluated on more exotic joint types as (1) we are not aware of any dataset that includes them, and (2) to ensure the highest chance of success for our baselines from prior works which have been trained on this exact dataset.

Notably, our evaluation is significantly more comprehensive – we test on all 46 object categories, whereas prior works typically evaluate on only five categories. This extensive evaluation reveals important limitations of existing approaches when handling out-of-distribution data (Fig. 5).

How does different VLM base models or the number of examples affect the performance?

Thank you for your suggestion! We have added an ablation study on the base VLM model (Fig. 11) and the number of examples (Fig. 10). The results demonstrate our robustness to the choice of VLM, and our ability to do in-context learning.

Iterative improvement examples

We have the quantitative results in Fig. 9 and qualitative examples in Fig. 4.

Dear reviewer VbEW,

Are you referring to the question about the number of prompting examples?

We have addressed this question under our second bolded section. In particular, we have included two additional experiments: robustness to VLM choices (Fig. 11) and the ablation with number of prompting examples (Fig. 10) as you have suggested.

Thank you for your thorough and constructive feedback! Here we addressed your comments:

How to prevent collision between part meshes?

Our API performs placement by aligning the child and parent's link centers along an axis and performing collision checks to ensure tight placement without intersection. Collision check was mentioned in line 196. We have added a more detailed explanation in line 247 to make it clearer.

More clarification on affordance extraction

Our system intelligently identifies which parts need articulation based on demonstrated interactions (Sec. 4.4). For example, given a video showing a bottom pane of a double-hung window being closed (Fig. 2, pane 3.5): (1) the VLM analyzes video to understand which part is being manipulated, (2) It then identifies the corresponding part in the 3D model's segmentation map, and (3) Only the relevant part (bottom pane in blue) is selected for articulation, rather than annotating all movable parts.

More clarification on the mesh retrieval process

The mesh results for text and visual inputs have the same abstraction. As Fig. 2 (pane 2) shows, the object is broken into individual part meshes. Even when a template object is retrieved in the case of visual inputs, the object is still broken into parts to ensure a unified API.

Why was retrieval used instead of mesh generation and can reconstruction method be computed?

Thank you for your suggestion! Although the initial results were obtained via retrieval, as we stated in the conclusion: our primary technical contribution lies in the articulation process, and our components can be integrated with any mesh acquisation methods. We have additionally included new results replacing the retrieval mechanism with meshes generated by a diffusion model. Please see our website and paper Appendix A.6. for the new results. We also quantify our superior performance using Chamfer distance in Table 2 (appendix A.5). Articulate-anything with mesh generation is far superior than all baselines. Even with retrieval, our mesh reconstruction quality is still better than prior works.

We'd also like to note that retrieval alone is sufficient for most applications. We have massive datasets of static 3D models (e.g., Objaverse with +10 million objects) but only minuscule datasets of articulated objects. Simply being able to tap into these static datasets and articulate them automatically would already unlock many potentials in robotics and other applications.

The robotic application is not solid enough. There is no real-world robotic results.

We have spent significant effort developing and setting up the control stack on our hardware robotic platform. Our experiment shows that articulate-anything can accurately generate simulated assets from real-world videos, which can then be used to train an RL agent in simulation. The trained policy can be deployed successfully on a real Franka arm, completing all tasks. In comparison, a manual annotation by domain experts using RialTo took humans an average of 40.58 minutes per task to complete. Please see our website for the video demonstration and refer to the updated Sec. 6 in the paper.

...
3. Give a realistic estimate of the final dimension of the object in the response. When assembled, your parts MUST stay within the final dimension estimate.
...
6. 
- Think CAREFULLY about how your parts contribute to the final dimensions of the object. Give calculation for each dimension.
- For example: If you intend on creating an object of size [1,0.25,2] from stacking two objects over the y-axis, you **SHOULD and MUST create two parts of dimensions [1,0.125,2] and [1,0.125,2] INSTEAD OF [1,0.25,2] and [1,0.25,2]**. Doing the latter is INCORRECT and would result in double the dimension along the y-axis.
...
7.
- Follow the examples to give justification for the dimensions of the parts, especially how they contribute to the final dimensions of the object body.
- E.g., vertically stacked parts should generally add up to the height of the object body/frame ect.
...

What are the definitions of manipulation tasks? How many objects per category are used in the experiments? How many trials are conducted on the same object? 

Dear Reviewer yrhN,

Thank you for your detailed response! We provide more detailed clarifications below. Please let us know if anything is still unclear and we’ll be happy to follow up.

W1. How are the detailed numerical parameters (e.g., 6-dof pose, size) of a part determined?

We will add this discussion along with relevant Python code to the Appendix.

Part positioning: Our system determines the positions between a child and a parent part by aligning the child and parent centers along a specific axis using their bounding boxes. For example, when placing a child "below" a parent:

# placing the child center directly below the parent's center along the z-axis
# x, y same as parent
target_position[2] += parent_aabb_min[2] - \
                child_aabb_max[2] - clearance

Collision check is applied afterwards.

Similar to prior works like URDFormer, we do not predict orientations. This design choice is motivated by a key insight: when all objects in the dataset are preprocessed to face the same canonical direction, the parts that should be combined together are already properly oriented. The real question is where to put them. For example, should the the suitcase handle be going on top or below the body?

Part sizing: For visual inputs, dimensions come directly from retrieved templates. For text inputs, our Layout Planner uses comprehensive prompting (detailed in our previous response) to ensure dimensional consistency.

W2. When a template object is retrieved in the case of visual inputs, the object is still broken into parts. This process feels unnecessary.

In articulated object modeling, breaking objects into parts is necessary for annotating fine-grained movements between parts. Joint prediction gives movement to otherwise static parts and is always required. The question then is: why must we also predict the parts' locations (link placement)? There are two primary reasons:

API consistency: For text inputs where we retrieve individual parts, link placement is clearly necessary. By applying the same part breakdown process to template objects, we ensure a streamlined and unified API across all input modalities - a point raised in the reviewer's earlier questions.

Context for detailed articulation: Consider the left and right doors in Fig.4b. These doors open differently: the left door opens outward from right to left, while the right door from left to right. For accurate joint prediction, we first ask the system to perform link placement (Fig. 2, pane 3, and Fig.4a). Then, we provide the link placement code for the VLM to fill in the joint code. This creates crucial context for more accurate articulation (e.g., if the link placement places the "door_1" to the left then joint actor knows that "door_1" variable explicitly refers to the left door in the Python code).

W3. Evaluation with CD.

The main link and joint prediction experiment is done on the PartNet-Mobility dataset while CD is computed on real-world objects. Since our system and some prior works (e.g., urdformer) retrieve objects while others (e.g., real2code) do not, it might be hard to ensure a fair evaluation for all baselines in PartNet. The true test for all methods is whether we can accurately articulate real-world objects where no gt meshes are available to any method so CD comparison on real-world objects is definitely a fair game for all methods.

We'd also note that CD is still a crude metric that does not take into account texture. We thus provide qualitative examples in Fig. 7 and the website. It's evident that our retrieved and generated assets are of markedly higher quality that the baselines.

W4. Object Manipulation Details in Robotic Application.

Thank you very much for the suggestion! We'll add these details to the Appendix:

• Task definitions and reward functions
• Per-object trial counts (5 trials per object) with different initial object poses
• One object per type
• Training details (3 random seeds, 2M steps each)
• Learning algorithm (PPO) and hyperparameters
• Policy architectures and obs/action spaces
• Hardware details

We’d appreciate it if you can let us know if there is any details we might have overlooked.

Q1. Link and joint annotations are masked out before being fed into our model. How to mask such annotations on the selected candidate?

To clarify: "masking annotations" means we remove all link and joint information from the retrieved URDF files, keeping only the geometric mesh data. This is in response to the reviewer's observation that PartNet-Mobility objects already contain human-annotated link and joint information. Our system then predicts spatial arrangements and joint parameters from scratch, without access to any human-annotated ground truth.

Thank you for your constructive review and continued engagement! We are happy to address any further questions or concerns.

W4 Robotic Details

As we can no longer update the paper with a revision, we will provide the text below. As before, please let us know if there is any details we might have overlooked.

Task definitions

We experiment with 4 tasks: closing a toilet lid, closing a laptop, closing a microwave, and closing a cabinet drawer. We first train a RL policy in simulation using Robosuite and Mujoco \citep{zhu2020robosuite, todorov2012mujoco}. The robot is a Franka Emika arm with 7 degrees of freedom and a gripper. The dense reward functions are manually designed to: (1) move the arm closer to the object (reaching reward), (2) reduce the object's target joint state (angle reward. e.g., making the toilet lid closed), (3) prioritize using the gripper and (4) penalize for sudden movements. The reward is provided below.

def reward(self, action=None):
	reward = 0.0
	if self._check_success():
        reward = 100
	elif self.reward_shaping:
        # reaching reward
        obj_pos = self.sim.data.body_xpos[self.obj_body_id]
        gripper_site_pos = self.sim.data.site_xpos[self.robots[0].eef_site_id]
        cur_dist = np.linalg.norm(gripper_site_pos - obj_pos)
        if cur_dist < self.closest_dist:
            self.closest_dist = cur_dist
            reaching_reward = 1
            if cur_dist < 0.1:
                reaching_reward = reaching_reward * 0.25
            reward += reaching_reward
            
        # angle reward
        if self.first_frame:
            self.first_frame = False
            self.prev_angle = self.sim.data.get_joint_qpos(self.targetted_joint)
        else:
            current_angle = self.sim.data.get_joint_qpos(self.targetted_joint)
            diff_angle = current_angle - self.prev_angle
            angle_reward = 0
            if np.abs(diff_angle) > 0.001 and self.sim.data.time > 3:
                if diff_angle < 0:  # Reward for closing
                    angle_reward = 2.25
                elif diff_angle > 0:  # Penalty for opening
                    angle_reward = -1
            reward += angle_reward
            self.prev_angle = current_angle

        # penalize if any joint is closer to the object than the gripper
        for joint_name in self.robot_joint_names:
            joint_pos = self.sim.data.get_joint_qpos(joint_name)
            joint_obj_dist = np.linalg.norm(joint_pos - obj_pos)
            if joint_obj_dist < cur_dist:
                joint_pose_penalty = -5
                reward += joint_pose_penalty
                break

        # penalize for large joint velocities
        joint_velocities = [self.sim.data.get_joint_qvel(
            joint_name) for joint_name in self.robot_joint_names]
        joint_velocities = np.array(joint_velocities)
        joint_velocities = np.abs(joint_velocities)
        if any(joint_velocities > 2) and self.sim.data.time > 0.5:
            action_penalty = -0.5
            reward += action_penalty

    return reward

Policy Learning: In simulation, we follow a teacher-student distillation training (\citep{eurekaverse, chen2020learning}). We first train a privileged state-based teacher using proprioception and object's pose and targeted joint's state. The policy is trained using the reward above with PPO \citep{schulman2017proximal}. We then distill a student policy that takes in depth frames from a side-view camera and proprioception via behavioral cloning. Visual inputs are encoded via a 2-layer convolutional neural network while non-visual inputs such as proprioception are encoded via a 2-layer feedforward network. The features are concatenated and fed to a 2-layer feed-forward network to produce actions.

Domain randomization: The physical parameters such as friction, stiffness, mass and damping are randomized. The camera viewpoints are also slightly deviated. The object's pose and scale are randomized. The initial target joint state is randomized (e.g., how far the drawer is already opened). The depth observations are augmented by adding Gaussian noise with random pixel blackouts (setting some pixels to 0). During distillation, we randomly delay the action and depth sensing by 60-100ms and the depth frames are updated at 30Hz.

Sim2Real Transfer: The policy outputs the position of each joint and a binary gripper command for closing and opening. The delta arm joints are limited to (-0.5, 0.5) for safety. The student policy is deployed zero-shot to a Franka Emika Robot with depth frames captured by a side Zed camera. The depth frames are center-cropped to reduce background noises. We use one object per task. Each task is rolled out 5 times with varied initial object pose and target joint state.

Dear Reviewer yrhN,

Thank you very much for your insightful questions! We address your questions below and are happy to follow up if anything is unclear.

W1

Part positioning

• Our VLM is only responsible for high-level planning while the API implements the precise positioning. We implement above, below, left, right, front, back, inside for the placement in similar fashion as the code snippet for below provided above. Note that without loss of generality, this allows a part to be positioned anywhere in the 3D space because (1) the API takes an optional clearance (offset) and (2) the python place_relative_to calls can be chained together.
• Thank you for the insightful observation. Indeed, the major problem with irregular meshes that we have encoutered is "loose" placement. In the left of this image, you can see that because the faucet's nose is concaving up, placing the spout above the body based on bounding boxes alone creates a large air gap. The solution is to perform iterative movement with collision checks to "ensure tight placement without intersection" (Line 248). Intuitively, this means just moving the faucet down the z-axis by a small increment until we detect a collision. Below, is the shortened python code. The iterative collision movement and check is the snap_to_place function. With this function, we ensure tight placement and small link error as shown on the right of the image.

def place_relative_to(self, child_link_name, parent_link_name,
	placement, clearance=0.0, snap_to_place=True):
	...
	if placement == "above":
		target_position[2] += parent_aabb_max[2] - \
                child_aabb_min[2] + clearance
            axis = 2
            direction = -1
    ...
    self.move_child_to_target_position(target_position)
    if snap_to_place and placement != "inside":
    	self.snap_to_place(...)

def snap_to_place(self, ..., max_iter=100, increment=0.01):
	collision = self.is_collision(child, parent)
	while not collision and i < max_iter:
		target_position = np.array([0.0, 0.0, 0.0])
		target_position[axis] += direction * increment
		... # move child to target position
		collision = self.is_collision(child, parent)

• snap_to_place stops as soon as a collision is detected. We have not encountered any case where a placement encounters collision right off the bat.

We will add these details and Python code to the Appendix in the camera-ready version.

Part orientation

We note that the part meshes in the dataset are extracted from some parent objects (i.e., a handle part always belongs to some suitcase object). Thus, if the objects are preprocessed to face some canonical direction, orientation between parts should not be an issue. In general, for a systematic treatment, a rotation degree argument can also be added to place_relative_to to control orientation while prompting the VLM accordingly but we have found this to be unnecessary.

W2 & Q1: Part Segmentation

We do not use link and joint annotations (no locations, no movement) but we do use ground-truth segmentations provided in the PartNet-Mobility dataset to break whole objects into parts. The availability of part segmentation is assumed in many retrieval-based works (e.g., Singapo, URDFormer, CAGE, NAP, ect)

As we responded to Reviewer bBwZ, in future works when applying our method to a larger dataset like Objaverse where part segmentation might be sparsely available, or to newly generated meshes from a diffusion model (Fig. 7, 22), an additional segmentation step is required. Particularly, in our supplemental mesh generation experiment, we first render multi-view RGBD images of the 3D model in simulation, and run SAM to obtain 2D segmentation masks. The 2D masks are lifted to 3D via projective geometry using camera parameters and depth maps. Once the 3D part segmentation is complete, we run our articulation method as described in the paper.

We will include these details to make it clearer in the camera-ready version.

W3 Chamfer distance on PartNet-Mobility evaluation

Thank you for this valuable suggestion! Our method requires no training instead relying on a few in-context examples (Fig. 10). So to make the comparison fairer, when evaluating each object, we remove that object from the candidate pool, preventing our system to retrieve the exact object. The result on PartNet-mobility, to be added to the paper, is included below (mean and std). The violin plot is included here.

Dear Reviewer yrhN,

W1: The x and y-axes are the same as the parent link when the placement is above and below (i.e. vertical movement). Similarly, for front and back, only the y-axis is changed (x, z are aligned). And for left and right, only the x-axis is changed (y, z are aligned). In the example, as the reviewer correctly pointed out, the x, y coordinates of part A and part B are indeed aligned (Part A is a child of part B in this case). The target_position is first set to the parent's center. Then, based on the placement, only one coordinate is changed. For brevity, we only provided a shortened code snippet in our earlier response. But we will add the full code to the appendix in the camera-ready version for clarity.

W4: Thank you for bringing this up! Take the task of closing a toilet lid for example. In simulation, the toilet lid is represented via a revolute joint ranging from 0 (fully horizontal) to 90 degree (fully vertically upright). The check success function involves checking if the joint state is small enough. For tasks like toilet, microwave, and laptop, the joint states are the angle of the lids and for the cabinet task, the joint state is the displacement of the drawer (how far is the drawer opened). For example,

    def _check_success(self):
        """
        Check if toilet lid has been closed.

        Returns:
            bool: True if toilet lid has been closed
        """
        toiletlid_angle = self.sim.data.get_joint_qpos("toilet_joint_1")
        return toiletlid_angle < 0.01

This allows automatic success checking in simulation for training policies. When deploying to the real-world, we monitor and record the trajectory, and determine success manually.

"Even with just text or image inputs, our success rates are still significantly higher than prior works." Is there any experiments for that? All what I can see is figure 6. Inferring from that, with text input, Link Placement has 68.6% successful rate and Joint Prediction has 48.9% successful rate. Those two time up to 34% success rate, which is only 10% higher than URDFormer in Figure 5? And in Figure 5, they are tested on all seen categories rather than just on largest category.

Can you just provide a single experiment comparing with baselines with fair input modality? (maybe just compare to urdformer to save time)

About "Mesh generation details", I think it's not trivial to lift multi-view 2D masks into 3D. Also, it's not reasonable when you have only the surface mesh (as we can see there's visual artifacts in your website drawer demo video). But I'll stop arguing this part since all previous works are also generated by retrieving object parts, and this doesn't hurt the major contributions of this work.

Thank you for your constructive feedback! Here we addressed your comments:

Retrieval

Our primary technical contribution lies in the articulation process. As mentioned in the conclusion, articulate-anything is general-purpose and can be integrated with any mesh acquisition method, be retrieval or generation. We have additionally included new results replacing the retrieval mechanism with meshes generated by a diffusion model. Please see our website and paper Appendix A.6. for the new results.

We'd also like to note that retrieval alone is sufficient for most applications. We have massive datasets of static 3D models (e.g., Objaverse with +10 million objects) but only minuscule datasets of articulated objects. Simply being able to tap into these static datasets and articulate them automatically would already unlock many potentials in robotics and other applications. Even with retrieval, our mesh reconstruction quality is still superior than prior works as shown in Table 2.

Technical contributions

Our work introduced three key innovations: (1) the ability to consume more diverse and grounded types of input, (2) modeling articulation as program synthesis, and (3) developing a closed-loop self-improving system. These advances significantly outperform prior work, achieving a sixfold increase in success rates (Fig. 5) with notably better qualitative results (Fig. 7). Our technical components provide substantial improvements over a standard LLM prompting baseline (real2code).

Our iterative refinement makes two important contributions: reliable failure detection (our critic highly correlates with ground truth, Fig. 15) - crucial for robotics applications - and a 2.4% improvement in joint success rate, which exceeds 25% of URDFormer's total performance. In safety-critical applications like robotics, simply being able to detect failure is important. Additionally, we currently use one camera pose for all tasks. Further camera optimizations (e.g., multi-view, zooming into small parts) can improve the actor's success even further

Other questions

• Type of inputs:

Our experiments use video inputs when comparing with other methods, as specified in Fig. 7's caption and the experimental details (line 790). While existing methods are designed for specific inputs (URDFormer uses cropped images, Real2code uses text bounding boxes), our approach uniquely supports multiple input modalities.

• Does CLIP scores based on text description mean that retrieved assets are not realistic?

As detailed in section 4.1 and Fig. 3, for visual inputs, we visually match real-world objects with simulated assets using a VLM. CLIP is used only for text inputs or initial object category filtering. For instance, in Fig. 3, the VLM selects the 3D model that best resembles the real-world monitor from available monitor assets.

• Can accuracy be reported across different categories

Thank you for your suggestion! We have added the accuracy breakdown across different categories in Fig. 20 and 21 in the Appendix. We use one camera pose for all tasks. As such, we have observed that smaller object parts are generally harder to place correctly, and objects with more esoteric movement are harder to predict joint, as the reviewer alluded to.

Thank you for your thoughtful questions! We address each of the questions below.

Different modality comparison

Our experimental results in Fig. 6 demonstrate strong performance across input modalities. Even with just text or image inputs, our success rates are still significantly higher than prior works. While this ablation is only run on the largest object category due to budget constraints, these results are further reinforced by the real-world results in Fig. 7. For real-world objects, our text and image-based methods succeed on some tasks whereas the baselines consistently fail.

Regarding the ease of data acquisation, our system handles casual, unconstrained inputs whereas prior works require heavy data curation as detailed in Appendix A.4. For example, as shown on our website, input videos are captured in cluttered environment (e.g., see laptop), with angled (e.g., window) and titled (e.g., microwave) views with no additional data cleaning. This unlocks the exciting potential of applying the system on internet-scale video datasets such as Ego4D or Youtube. In contrast, URDFormer required manual bounding box correction (Fig. 18), and Real2Code required extensive scene setting, scanning via a Lidar-equipped phone, tuning segmentation masks and hyper-parameter (Fig. 17).

Mesh retrieval on Objaverse

Object labels are not a requirement of our approach -- we used them solely as an optional filtering step to reduce computational costs whenever possible. Our system can leverage standard efficient large-scale visual retrieval algorithms (e.g., approximate nearest neighbor, hierarchical embedding, locality-sensitive hashing) that enable search engines to search through billions of images. Secondly, having part meshes is also not a requirement. Our method naturally handles both whole objects and their constituent parts, as demonstrated in our mesh generation results, explained in more details below.

Mesh generation details

After obtaining a static mesh, we perform a standard 3D segmentation technique to obtain part meshes. Particularly, we first render multi-view images of the 3D model in simulation, and run SAM to obtain 2D segmentation masks. The 2D masks are lifted to 3D via projective geometry using camera parameters and depth maps. Once the part meshes are obtained, we apply our method as described in the paper.

Please let us know if you have any more questions that we can address!

Dear reviewer bBwZ,

We've included an additional experiment in Appendix A.7 (Fig. 23) as requested. Please see the revised draft. To save time, this ablation was run on the corresponding seen classes of Real2Code and URDFormer. Articulate-Anything's text and image-based, while degrading compared to our main video-based method, still significantly outperforms the comparable baselines.

We'd also like to note that "Those two time up to..." should not have been done. The joint prediction success rate already factors in the link failure (Line 308). As you can see from the left-most pane of Fig. 8, joint prediction failures already include link failures.

Dear Author,

The unfair comparison part has been well addressed. Hopefully the author could state that clearly in the camera ready version.

One of my other concern is about the technical contribution. The major component of this paper seems to be the iterative refinement part. While it is 2.4% growth in the success rate, it seems to be subtle comparing to your ablation score (over 70%).

Also, the success thresholds for link placement and joint prediction are also quite weird: 0.25 rad is large enough to make a difference. Is that one of the limitation of this work since it's hard for VLMs to sense smaller changes?

In all, I personally still don't think prompting VLMs to place retrieved meshes and assign joints has enough technical contribution. However, that might not be the community standard. Thus, I raise my score to borderline accept and will let other reviewers and AC to judge that.

Dear Reviewer bBwZ,

We sincerely thank you for your thorough review and constructive suggestions throughout this process.

We will include, if space permits, or at least reference the new modality results in the main text.

Regarding your concern about thresholds, we conducted additional analysis with a much stricter threshold of 0.01 radians (compared to the original 0.25):

Even with this significantly stricter threshold, our method maintains a 3.5-6× improvement over baselines. Most failures at this threshold occur with very fine-grained and small movements (e.g., keyboard buttons). These errors are typically not discernible to human eyes at our current camera pose.

Previously, the reviewer acknowledged that "There are enough technical contributions for this work." So we would like to re-iterate our contributions beyond iterative refinement:

• State-of-the-art performance: Our method achieves sixfold improvement in quantitative metrics (Fig. 5) with substantially better qualitative results (Fig. 7).

• Multi-Modal Flexibility: We uniquely support in-the-wild inputs through text, images, and videos, eliminating the need for specialized data like point clouds or connectivity graphs.

• Flexible Asset Generation: Our approach works with both retrieved and generated meshes (Fig. 7, Table 2), producing high-fidelity digital twins that closely match real-world objects.

• Scientific Insights: Not only have we shown that the method is highly effective, through extensive ablation studies, we have provided scientific insights:

  • In-context learning: Our training-free approach, as you noted, eliminates "the need for large-scale datasets, which are challenging to obtain for articulated objects." Through a few in-context examples alone, we achieve dramatic performance improvements from 4.6% to 77.7% (Fig. 10).

  • Richer input modality: Our system leverages richer input modalities to improve accuracy from 48.9% to 77.7% (Fig. 6). This capability enables crucial disambiguation—for example, while a window door could theoretically use either a revolute or prismatic joint, video input allows our system to determine the correct joint type (Fig. 7).

  • High-level API: Rather than directly predicting numerical values like previous approaches, we leverage high-level planning strength of VLM while delegating low-level details like kinematics to a classic pipeline. This architecture demonstrates superior performance against baselines even when using identical input modalities (Fig. 23).

  • Self-evaluation and correction: We introduce the first self-evaluating articulation system that can detect failures — a critical feature for robotics applications. This capability enables autonomous refinement, yielding significant improvements in both link placement (5.8%) and joint prediction (2.4%). While the joint gain is more modest, it is because joint prediction task is harder. More camera pose optimization is likely to increase this gain.

  Thank you again for your careful review and consideration!

Dear Reviewer bBwZ,

Regarding the critic component, as we detailed in two previous responses, this component uniquely enables us to detect failures and improves the system. The improvement is in excess of 25% of URDFormer's performance. We believe further camera pose optimization (e.g., multi-view, zooming into small parts) will increase the gains even further.

As we stated in our contributions in the introduction and our rebuttal, our main technical contribution is in the articulation process. We consider mesh acquisition front-end to be secondary as our method can work with either retrieved or generated assets. We will add more details on the generation process to the paper as suggested.

Lowering the threshold was done on the main result. We have included an ablation on comparable modality as suggested in Fig. 23. Nonetheless, we believe that videos is the correct format to describe motion. Conceptually, the window example in Fig. 7 demonstrates that impoverished input modalities fundamentally cannot capture the necessary information for joint articulation in all cases. Further, as argued in our first response, we believe our method to be much more easily and automatically deployable to large-scale datasets as prior works require heavy manual curation.

Once again, we thank you for your constructive review, perspectives and for raising our score. Your suggestions have helped strengthen the paper, and have been incorporated into the revision.

Best regards,

Authors

Dear Reviewer bBwZ,

As the rebuttal period draws to a close, we'd like to thank you for your thoughtful review and valuable suggestions, which have helped strengthening our paper. Below, we address the points that you raised.

Regarding the performance improvement: Your observation about comparing against previous works is well-taken. Upon closer inspection, we note that 14.7% of joint failure originates from incorrect link placement in the previous stage (Fig. 8). Thus, the maximum achievable joint accuracy is only 85.3% (not 100%). At the end of iterative refinement, our accuracy is at 75%, about 10% shy of the ceiling. The remaining failure cases are hard; they tend to involve small objects with irregular movement like Pliers and Knife (Fig. 21). In these cases, if the actor fails to correctly articulate the object in the first iteration, even the critic mechanism proves insufficient for correction. We believe that enhanced camera optimization strategies—such as multi-view systems or dynamic zoom capabilities for small objects—could address these limitations, but reserve it for future work.

Regarding segmentation: as the reviewer previously suggested, most other works (e.g., Singapo, URDFormer, CAGE, NAP) requires part segmentation: which can either be already available in the dataset (e.g., PartNet) or predicted (e.g., Real2code). Our main result is based on retrieval, leveraging existing part segmentations. For our supplemental results with newly generated meshes, we predict segmentations using SAM, similar to Real2code's approach. We explained the 3D segmentation procedure in our earlier response. We will add these implementation details to the camera-ready version.

Regarding video inputs: We agree that videos might be harder to acquire. However, we argue that richer input modalities, whenever available, should be used for improved accuracy (Fig. 6). Additionally, our system also supports images and texts, providing flexibility to the user.

In terms of practicality, our system offers two key advantages: (1) working with casually captured in-the-wild inputs out-of-the-box, and (2) automatically detect failures (via a critic). These features are important as otherwise, the cost of hiring humans to first extensively curating the data (e.g., manually correcting part bounding boxes, Appendix A.4, Fig. 17-18) and afterwards filtering out incorrect predictions might outweigh the benefits of automatic articulation. Without such automation, manual asset annotation might indeed be more cost-effective.

Thank you,

Authors

Thank you for your positive review! We have updated our draft to incorporate your feedback.

References to other actor-critic VLM/LLM works

Thank you. We agree that a more detailed discussion of our differences from previous works concerning actor-critic is warranted. The text2reward paper that the reviewer mentioned uses human feedback while our system utilizes a VLM as a critic. This is a very important distinction as it allows the system to be fully automatic and scales with data without any human effort. Some papers that automate the verification process such as Code as Reward and Kwon et al 2024 employ very domain-specific heuristic such as object tracking. There are a few papers that use VLM as a success detector for robotic task such as Du et al 2023 and Guan et al 2024, providing a binary reward or natural language feedback. However, these are essentially critic-only systems. To our knowledge, we are the first to integrate both VLM actor and critic in a closed-loop system, enabling self-correction over subsequently iterations, all without human efforts.

Lack of statistical confidences

We have added standard deviation for continuous errors and 95% CI for binary errors to Table 1 (Appendix A.5), 95% CI to Figure 5, and a violin plot in Figure 19 in the Appendix. As before, our superior performance lies far beyond the margin of error.

Robotics and in-the-wild application

We have spent significant effort developing and setting up the control stack on our hardware robotic platform. Our experiment shows that articulate-anything can accurately generate simulated assets from real-world videos, which can then be used to train an RL agent in simulation. The trained policy can be deployed successfully on a real Franka arm. In comparison, a manual annotation by human domain experts using RialTo took an average of 40.58 minutes per task to complete. Please see our website for the video demonstration and refer to the updated Sec. 6 in the paper. For in-the-wild results, we have also added the results integrating our system with a mesh generation model to produce more customized assets. Chamfer distance showing superior mesh reconstruction quality is also included in Table 2.

Other questions

• Weighting the loss: as we wrote in Sec. 3, the losses are approximated using a VLM critic. We use few-shot prompting i.e., no explicit training or fine-tuning of the losses.
• Success criteria: we use a very small threshold as the success criteria. We've found that this threshold is so small that small errors are not noticeable to the human eye. But we also provide the raw errors (unthresholded) in Table 1 where Articulate-Anything is still much better than all baselines.

Dear Reviewer SpUi,

Thank you for your positive feedback! The RL policies trained in simulation are directly transferred to the real arm where the tasks are fairly simple and deterministic. We trained 3 random seeds per task for 2 million steps and selected the best checkpoints. We run 5 roll-out trials per task with slightly varried object poses. All of these policies are successful in the real-world. Those simulated policies are also included on the website for cross comparison with the real-world execution.

Besides the quantitative measure of the human annotation cost in Fig. 12, we are also trying to include more baselines for the camera-ready version: particularly, training the same RL policies using assets generated by URDFormer and Real2Code. We do expect these baselines to perform much worse given the poor quality of generated assets (Fig. 7). These results are not currently available because whereas our method can articulate real objects out of the box, other baselines require heavy curation (Appendix A.4) so it's difficult to curate and include them in time on top of all other experiments we've included in the global response to all reviewers. Once these baselines are finished, we will include a quantitative comparison table between these 4 baselines (ours, human-annotated, Real2Code, URDFormer) that the reviewer might expect.

Please feel free to let us know if there's any more analysis or experiments that can be reasonably run before the deadline that you'd like to see to improve your rating or confidence.

Thank you.