Data Scaling Laws in Imitation Learning for Robotic Manipulation

We thank Reviewer SnY8 for their insightful and positive review.

Generalisation performance is only discussed for policy learning in the single-task setting, the paper doesn't consider the multi-task setting which remains very relevant in modern robot learning applications.

Yes, as we discussed in the limitations section of the paper, we do not consider task-level generalization at this stage. We believe that addressing this would require collecting vast amounts of data from thousands of tasks, which is beyond the scope of our current work. We hope to explore task-level generalization in the future when more resources become available.

The scale of the demonstration datasets is limited, more work is required to ensure that the scaling laws hold true at larger scales.

Thank you for bringing this up. Compared to the data used in current large language models, our dataset is indeed relatively small. In our future work, we will verify our conclusions on larger datasets and a more complex set of tasks.

Generalisation performance is only discussed with respect to two criteria, the authors also argue that these criteria encapsulates all factors a policy may encounter in real-world deployment. In general, I would say that this argument is incorrect.

Thank you for this insightful suggestion. Our previous statement may indeed have been too absolute. Our intention was to convey that for most single tasks (excluding task-level generalization for now), an ideal robot policy deployed in the real world should handle new environments and objects. Our paper primarily studies environments and object generalization, but there are many other factors we have not considered—for example, motion-level generalization is needed when the environment contains obstacles. If humans are present, we need to consider the safety of interacting with them and the robustness to various human disturbances, and so on. We also look forward to hearing your thoughts on generalization.

The authors argue that they focus on out-of-domain generalisation, however they do not quantify out-of-domain in their experiments. This is more generally a challenge for evaluating generalisation performance of machine learning experiments. It remains a weakness of the current approach as we aren't quantifying out-of-domain which makes it challenging to comment on generalisation performance. With this being said the appendix provides visuals of the objects and environments which provides sufficient evidence for data diversity to support the authors claims.

Thank you for pointing this out. We fully agree with your perspective that quantifying out-of-domain generalization is a challenge faced by all machine learning experiments, especially in in-the-wild experiments like ours. Therefore, we define out-of-domain as the policy’s ability to work on unseen instances in both the environment and object dimensions. Here, “unseen” does not refer to only changing partial layouts or settings of the environment, or partial attributes of the objects, but rather to working in entirely new environments (e.g., a new restaurant) and with new objects (e.g., a newly purchased cup). As you have observed, we provided images of the environments and objects in the appendix to support this point.

The scoring rubric for tasks has the potential to introduce subjective bias. In spite of this potential weakness, the authors effectively argue the need for this rubric versus more rudimentary metrics such as MSE. The results of this work hinge on a readers confidence in the scoring rubric for experiments, in general I feel confident in the quality of this work, however, it is important to mention this point.

Thank you for this important observation. The evaluation metric is indeed another challenge in robot learning. We believe that the scoring rubric captures more nuanced behaviors and reflects the robot’s real performance, but it inevitably introduces subjective bias. We will make this point clearer in the final version of the paper.

We thank Reviewer facp for their thorough and insightful review.

Considering the multiple other factors that can affect learning performance, the authors should provide data details for each environment, object, and environment-object variation. This could include specifying the number of demonstrators [1] (with their IDs), listing the data collection protocol [2], and showing relevant statistics (e.g., the number of failed demos, action variance [3], and task horizon for each demonstrator [3]). Such information would clarify that any observed variation is due solely to the environment, object, or environment-object pair rather than bias in the data itself.

Thank you for your insightful suggestion. In the table below, we have listed statistics of some data. Specifically, we present the data for the tasks of Pour Water and Mouse Arrangement, which were used in the experiments on generalization across both environments and objects, and for the tasks of Fold Towels and Unplug Charger, which were used when verifying the data collection strategy. The table includes the total number of valid demonstrations (after SLAM filtering), the number of valid demonstrations collected by each collector (with their IDs), and the average time horizon for each task.

We have provided the protocol instructions given to data collectors here: Data Collection Protocol. We will release all data, code, and models associated with our work. We hope this information helps clarify any concerns regarding data variations.

One concern about particular data collection hardware (i.e., UMI) is that it may introduce added bias. For instance, if a specific teleoperator is highly skilled, they might gather higher-quality data by using optimal speed movements or demonstrating behaviors at the precise height where the robot is positioned. The authors could clarify this in relation to the above point.

Thank you for bringing this to our attention. We have also noticed that data quality significantly impacts the policy’s performance. Therefore, we ensured that our data is of high quality and that the behaviors are consistent. Before collecting the final data used in our experiments, we made sure that all four data collectors were highly skilled, and we provided detailed instructions for each task being performed (see our previous response). Ensuring high-quality and consistent data is manageable with only four data collectors. However, we recognize that scaling up to a larger number—say, 100 data collectors—would make it challenging to guarantee the same level of data quality and consistency. Exploring how data of varying quality affects the policy and how to learn from such highly multimodal data is a very important issue, which we leave for future work.

Thank you to the authors for the thorough responses. My main concerns, particularly about data consistency, have been effectively addressed. I have updated my score to "accept."

We thank Reviewer GM4F for their insightful and positive review.

The analysis focuses mainly on two relatively simple tasks (pouring water and rearranging a computer mouse). While the paper would benefit from investigating more tasks and also including more challenging tasks, I am aware that it is quite an effort to set up additional experiments with real robots and collect large training sets.

Thank you for bringing that to our attention. We acknowledge that the tasks investigated in the paper are limited in both number and difficulty. This is mainly due to our use of UMI as the data collection device. The current version of UMI has certain constraints, as discussed in our paper, which prevent us from collecting data on tasks involving occlusions in the camera’s view (e.g., opening doors or drawers) or those requiring very precise actions (e.g., inserting objects). We hope to address these limitations in the future by using a more advanced data collection device that will allow us to validate our data scaling laws on a larger and more complex set of tasks.

The objects considered in the tasks are always objects of the same category (e.g., water bottles) and, thus, very similar. I believe that this is the reason why increasing the number of objects per environment in Figure 6 does not improve the task performance much. Essentially, the objects are always quite similar, while the environments differ quite significantly, and therefore, it is more important to collect data from more environments rather than with different objects in the same environment.

This is an interesting perspective, and we believe it is quite reasonable. Because objects of the same category are relatively similar in shape, they are easier to generalize over, which is why increasing the number of objects per environment does not significantly improve task performance. We look forward to exploring more challenging generalizations in the future, such as generalizing to novel (but related) object categories.

In Figure 6, sometimes increasing the number of objects per environment decreases the task performance quite considerably (e.g., right plot M=4). This seems quite counterintuitive. What could be the reason for this?

There could be two reasons for this: (1) Training Randomness: It’s possible that using a different random seed could lead to improved performance. (2) Evaluation Noise and Scoring Bias: The evaluators may have scored this policy lower. We believe that, theoretically, the performance did not decrease, but due to various random factors—especially since we did not perform multiple seed repetitions (complete real-world evaluations are very complex and time-consuming, so we only did one seed)—some scores may have noise. We recommend focusing on the overall trend shown in the figure: as the number of environments increases (e.g., to 16), the performance gap between collecting multiple objects per environment and just a single object becomes negligible.

What is exactly is the correlation coefficient r (e.g., line 380)?

If two variables $Y$ and $X$ satisfy the relation $Y = \beta \cdot X^\alpha$ , they exhibit a power-law relationship. Applying a logarithmic transformation to both $Y$ and $X$ reveals a linear relationship: $\log(Y) = \alpha \log(X) + \log(\beta)$ . Therefore, to determine whether the two variables $Y$ and $X$ satisfy a power-law relationship, we can transform the original data by taking the logarithm and fit a simple linear regression model to the transformed data. We use the correlation coefficient $r$ to assess the quality of the linear fit; if the absolute value of $r$ is close to 1, it indicates that the log-log plot of $Y$ and $X$ fits well with a straight line, suggesting a power-law relationship.

There are no standard deviations in the plots. Consider adding the standard deviations to make it easier to assess the empirical significance of the results.

Thank you for this valuable suggestion. In Table 1 of the original paper, we reported the standard deviation across 8 unseen environments. We have now uploaded a revised version of the paper, where all the curve plots in the main text include shaded regions representing the 95% confidence intervals to make it easier to assess the empirical significance of the results.

Caption 7: "In the setting where we collect the maximum number of demonstrations, we examine whether the policy's performance follows a power-law relationship with the total number of demonstrations." I believe the first "demonstrations" should be something like "environment-object combinations".

Yes, your understanding is correct. This setting indeed has the maximum number of “environment-object combinations” (64 combinations) as well as the maximum number of demonstrations, totaling over 6,400. Therefore, we chose this setting to study how changes in the number of demonstrations affect performance.

We thank Reviewer xsiU for their insightful and positive feedback.

The primary issue with the paper is that it does not mention the initial conditions for the robot and the environment. To understand what the authors mean by "90% success", we must have a good sense of what the task and environment variations look like.

Thank you for bringing this up. In fact, in Appendix D: Task Details, we provide a detailed description of the initial conditions for the environment and objects for each task. Generally speaking, during every evaluation, we command the robot to move to the same initial joint position (primarily for ease of testing), but there are significant variations in the environment and objects. Firstly, to test generalization, our environments or objects are ones that the policy has never seen during training. Secondly, the initial positions of the objects vary greatly, covering almost the entire workspace within the robot's kinematic reach. Below, we provide descriptions of the initial conditions for each task:

1. Pour Water: The bottle is randomly placed on the table, provided it is within the robot’s kinematic reach. The relative initial position of the bottle and mug is also randomized, ensuring they are spaced variably while keeping the mug visible to the camera after the bottle is grasped. Only the red coaster has limited initial conditions; it is a 9 cm diameter circle consistently placed approximately 10 cm to the right of the mug, used across all environments.
2. Mouse Arrangement: The mouse can be positioned anywhere on the table, as long as it remains within the robot’s kinematic reach. The mouse may be oriented straight ahead, in which case the robot needs to grasp it directly from behind. Alternatively, it might be slightly tilted to the left or right, necessitating the robot to employ non-prehensile actions, such as pushing the mouse into the correct orientation before closing the gripper for picking it up. In each test, the relative position between the mouse pad and the mouse is also varied, with the mouse pad randomly placed around the mouse.
3. Fold Towels: The initial position of the towel may vary on the table, provided it remains within the robot’s kinematic reach and its tilt angle relative to the table’s edge does not exceed 15 degrees. We assume the towel has already been folded several times.
4. Unplug Charger: The charger and power strip can be placed anywhere on the table, provided they remain within the robot's kinematic reach.

The authors don't create much of a variation in the task difficulty – as a result it is hard to tell if the scaling laws derived in the paper scales in a similar way with harder or easier tasks.

Thank you for pointing this out. There are indeed some differences in difficulty among the tasks we have selected. For example, Pour Water is a relatively long-horizon task and is more challenging than Mouse Arrangement. However, we did not use even more difficult tasks mainly because the current version of UMI has certain constraints, as discussed in our paper, which prevent us from collecting data on tasks involving occlusions in the camera’s view (e.g., opening doors or drawers) or those requiring very precise actions (e.g., inserting objects). We hope to address these limitations in the future by using a more advanced data collection device that will allow us to validate our data scaling laws on a larger and more complex set of tasks. We believe that the trends regarding data scaling laws also hold true for extremely hard tasks.

The "new tasks" that the authors collect data for and evaluate in (towel folding, unplug charger) does not seem to be of a similar level of difficulty as the primary tasks talked about in the paper.

The reason for not selecting more difficult "new tasks" is the same as mentioned above. We look forward to utilizing better data collection device in the future to validate our data scaling laws on a larger and more diverse set of tasks.

Why do we see the normalized score drop at the very end as the number of demonstrations are scaled?

Thank you for pointing this out. We believe there are several possible reasons: (1) The model size might not be sufficiently large, lacking enough capacity to fit the increased data volume. (2) The training time might be insufficient, meaning the model has not fully converged. (3) Training randomness—it's possible that a different random seed could lead to improved performance. (4) Evaluation noise and scoring bias are unavoidable. Given that a full evaluation is time-consuming (see our next response), we have yet to determine the precise cause. We lean towards believing that the theoretical performance of the model remains unchanged, and that the slight drop in score may be attributed to noise during training and testing.