PokeFlex: A Real-World Dataset of Deformable Objects for Robotics

• Q1 - Origin of ground truth mesh. Yes. As mentioned in L091 -094, "we leverage a professional multi-view volumetric capture system (MVS) that allows capturing detailed 360° mesh reconstructions of deformable objects over time (Collet et al., 2015)". To further clarify this, we updated the manuscript as follows:

  "we leverage a professional multi-view volumetric capture system (MVS) that allows capturing detailed 360° mesh reconstructions of deformable objects over time (Collet et al., 2015), **which we use as ground truth meshes**." 
  

• Q2 - Use of transparent stick for poking: Currently the robot data includes the force in the contact point and the position of the contact point. The fields in the robot data will be extended to include the origin of the tool, and the kinematic configuration of the robot per time step. With that information, any user of the dataset can easily generate a point cloud or a mesh for the poking stick (length 192 mm, radius 10 mm), if needed. However, we note that the transparent stick does not appear in the raw depth images nor in the raw point clouds recorded by the Azure Kinect cameras. For clarity, we updated the manuscript with the following description in L188-190:

   “The dataset also provides the CAD model for the mounting tool, which holds two RealSense cameras and a 192 mm long acrylic stick with a radius of 10 mm.”
  

• Q3 - Accuracy of force measurements: : Prior to the data collection we validated the accuracy of the force measurements of the UR5e robot using a force Sensor (Bota MinieOne Pro) which has a 40mN resolution in the vertical/poking direction. The results of such a test can be seen in the Plot 1 below. Note that the error of the force sensing from the UR5e only becomes large when applying large forces. Otherwise, the UR5e robot demonstrates sufficiently reliable force readings.

  • Plot 1: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_force_accuracy.pdf

• W1 - Other applications of the dataset and partial observations: Regarding other applications see general response 2. Moreover, we note that the models presented by Pokeflex deal with partial observations. The sequences of images from a single viewpoint, or the point cloud data from the Kinect sensor are indeed partial observations, from which the presented models are able to recover deformed 3D meshes given a template mesh.

• W2 - Large scale evaluation and Related work: The paper never claimed to be a tool for large-scale evaluation. PokeFlex is a smaller, high-quality dataset that can indeed be used for fine-tuning as the reviewer suggests. Regarding the related work presented in GarmentLab, we remark that the paper was uploaded to Arxiv, on the 2nd of November 2024, one month after the ICLR deadline. As such it was not possible for us to include that reference in our related work. The objects in such a dataset fall in the category of static scans, and as we mentioned in L86-L89, statics scans are unsuitable for capturing the temporal dynamics of objects undergoing deformations. We further note, that, to the date of publishing this response, GarmentLab only provides 3 real-world static scans of plush toys, as detailed in their appendix and link in their GitHub repo (https://drive.google.com/drive/folders/1CqJILIK8VQ-RCuLa_aFN-WtYTbovpFga Assets/ReaWorld). In contrast, we provide multimodal time-sequence recordings of 17 volumetric deformable objects under active deformation. The latest version of our paper now includes Garmentlab in the related work.

• W3 - Number of object and object types: . The dataset consists of objects with different levels of difficulty, going from simple geometric shapes (primitives) to complex 3D geometries like the octopus, the turtle, the Stanford bunny, and the 3D-printed heart. The decision to include simpler geometries allows the development of new techniques methodically, i.e., identifying material parameters for the foam dice should be easier than for more complex objects like the Stanford bunny. Finally, while we recognize that the Teddy bear from GarmentLab is a complex object, we remark that Pokefex also includes complex plush toys such as the octopus and the turtle.

• W4 - Action selection: Extending our comments in the general response 4, we remark that PokeFlex consists mostly of poking sequences and we argue that the dropping sequences are also quite relevant. Consider the work in [1] that evaluates a method to learn neural constitutive laws on two volumetric deformable objects during dropping. Taking the octopus as an example, we agree that the object does not deform significantly during free fall, however, the object also exhibits large deformations as soon as it makes contact with the floor, as can be seen in the last frames of the sample sequence provided. As such, further extensions of works like [1] can easily benefit from the dropping sequences of PokeFlex.

  • [1] “Pingchuan Ma et al, Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics”

• W5 - Positioning wrt to deformable object simulation and sim / real differences: While the deformable object simulation has been significantly improved on IsaacSim, the sim-to-real gap is still large. Not even the official developers of IsaacSim can comment on the general “sim-to-real” gap for deformable objects right now. See discussion on IsaacLab repo. https://github.com/isaac-sim/IsaacLab/issues/723 . We are currently in contact with several simulation teams from both academia and industry to evaluate potential ways in which our dataset can help tune their simulators. A thorough analysis of volumetric deformable object simulation compared with the real-world behavior captured by our dataset is then out of the scope of this submission and left as future work.

• W6 - Action protocols. See W4.

• W2 - Reason for using an acrylic stick and its implications on ToF-depth readings (Line 4-7): The acrylic stick was chosen to minimize occlusions on the object, leading to better reconstruction quality. In a previous setup, we used a Robotiq gripper for poking, which resulted in lower-quality reconstructions and the loss of texture in the deformed region, due to occlusions. We do not observe any impact on the Azure Kinect’s depth readings caused by the stick, as it does not appear in any point cloud visualizations.

• W3 - No experiments for dropping (Line 8-10): While we currently lack experiments using the dropping data, we believe these sequences are valuable due to their higher reproducibility compared to poking (Any user of the dataset can drop objects without the need of a robot). Additionally, they hold potential for fine-tuning simulators and extending existing works, such as [1].

  • [1] “Pingchuan Ma et al., Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics”

• W4 - Combination of modalities only for images and robot data (Line 10-12): We are actively running new experiments that include the combination of point clouds and robot data. These results will be included during the rebuttal period.

• W5 - Choice of thresholds (Line 12-14): The thresholds were determined heuristically based on experimental observations.

• W6 - Incorrect citation format (Line 14-15): Thank you for pointing this out. We have corrected the citation formatting in the updated version of the paper.

• W7 - Non-distributable license for the pizza object (Line 15-17): We are in contact with the authors of Pizza model to address this issue. For now, we have removed the pizza from the dataset and all related experiments for the rebuttal period. For the camera-ready submission, we plan to use our own pizza model. We have already 3D printed a first version (See video 1).

  • Video 1: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/videos/new_3d_printed_pizza.mp4

• W8 - Dataset not fully released (Line 17-21): The full dataset will be released by the camera-ready deadline (upon acceptance), as is common practice in other major conferences. The dataset is >500GB, and while we have the capacity to release such large datasets, current restrictions prevent us from releasing it without violating the double-blind policy. As stated in the paper, we are committed to making the entire dataset publicly available.

• Q1 - Choice of epsilon: The epsilon value was set to 0.4, as this provided a good overlap between the region of interest and the poking region of many objects.

• Q2 - Choice of values in Equation 3: The condition in Equation 3 ensures that the bottom surface of the object is not included in the region of interest, which can occur for thin objects like the sponge. As stated in L359, we scale the meshes to a unit cube, which justifies the value of 0.2 aiming to exclude the bottom surface while including the deformed region.

• Q3 - Choice of weight for LRoi: The weight for LRoi was selected heuristically to balance its contribution to the overall loss function.

• Q4 - Train-validation split: For multi-object training, we randomly selected one poking sequence per object for validation, with all other sequences used for training. All evaluations are based on the extracted validation sequences.

• Q5 - Experiments for all objects: Additional experiments now include all objects (see general response 3).

• Q6 - Including more objects in Figure 11 and training setup: We have generated equal graphs as Figure 11 for all objects, along with a histogram showing the sample distribution for each object (see Appendix A.6). For the training setup, the dataset contains only samples in deformed configurations.

  • Appendix A.6: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA6.pdf

• Q7 - Distribution of samples: We now provide insights into the distribution of the data using histograms (see Appendix A.6)

  • Appendix A.6: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA6.pdf

• Q8 - Evaluation under different lighting conditions: Environmental conditions were not explicitly varied during data acquisition since consistent lighting is integral to the MVS system for proper reconstruction. However, we introduce new experiments to evaluate the model’s robustness to modified lighting (see Table 3 in W1). Lighting modifications were achieved by adding or subtracting a constant value, and introducing random noise to all image channels. We applied the modifications to the RealSense images and evaluated on the model that was trained on the ground truth RealSense images.

We have extracted a total of 8 weaknesses from your text. In the following we would like to address these and answer your questions:

• W1 - Results show only a subset of the images (Line 1-4): We have updated our results to consistently include all objects in the experiments (See tables below and general response 3). Additionally, we provide detailed results for all objects for several modalities (see Appendix A.5).

  • Appendix A.5: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA5.pdf

• Table 3:

  Input	$\mathcal{L}_{\text{PFD}} \cdot 10^3 \downarrow$	$\mathcal{L}_{\text{ROI}} \cdot 10^3 \downarrow$	$\text{RPFD} \downarrow$	$\text{CD}_{\text{UL1}} \text{[mm]} \downarrow$	$J(\mathcal{M}_{\text{P}}$, $\mathcal{M}_{\text{GT}}) \uparrow$
  Volucam	6.69	8.38	0.689	7.433	0.799
  RealSense	7.91	7.37	0.693	7.675	0.806
  Kinect	11.20	9.79	0.839	8.505	0.767
  Rendered	12.97	12.28	0.826	8.761	0.761
  Brighter RealSense	14.57	13.25	0.853	9.126	0.754
  Darker RealSense	13.89	11.74	0.957	9.831	0.736

• Table 4: | Input | $\mathcal{L}_{\text{PFD}} \cdot 10^3 \downarrow$ | $\mathcal{L}_{\text{ROI}} \cdot 10^3 \downarrow$ | $\text{RPFD} \downarrow$ | $\text{CD}_{\text{UL1}} \text{[mm]} \downarrow$ | $J(\mathcal{M}_{\text{P}}$, $\mathcal{M}_{\text{GT}}) \uparrow$ | |------------------------------------------|------------------------------------------------------|------------------------------------------------------|------------------------------|-----------------------------------------------------|-------------------------------------------------------------------------| | Images | 6.69 | 8.38 | 0.698 | 7.43 | 0.799 | | Robot data | 7.43 | 5.80 | 0.847 | 8.01 | 0.785 | | Images + robot data | 5.39 | 5.10 | 0.594 | 6.64 | 0.821 | | Dense synthetic point clouds (5k points) | 4.76 | 4.92 | 0.569 | 6.33 | 0.831 |

• Q1 - Synthetic point clouds: For the dense point cloud setting, we set the number of points to 5K as it is sufficiently dense to represent the geometry of the objects, while not being too big to hinder the inference speed of the model under 100Hz on an RTX4090. The sparse synthetic point cloud was set to 100 points to stress test the reconstruction method's performance since this is the regime where the quality of the mesh reconstruction starts degrading significantly.

  When using synthetic point clouds, we attribute the imperfect reconstruction results to limitations in the NVP architecture that we used. Such architecture excels at capturing global deformation. However, it has a harder time reconstructing smaller / local deformations as the ones caused in the poking area.

• Q2 - Real-NVP architecture: The conditional Real-NVP architecture is referenced in L240-241 and it is an architecture that given a template mesh outputs the per-vertex displacement to match a deformed mesh configuration while preserving the original mesh topology. For clarity, we added a brief description of Real-NVP architecture in L242-245 as follows:

  "Real-NVP utilizes a series of conditional coupling blocks, each defined as a continuous bijective function. This continuous bijective operation ensures that the model is homeomorphic, which allows stable deformation of a template mesh while preserving its topology."
  

• W1 - Position wrt to PLUSH: After careful reconsideration of the PLUSH dataset, we note that such a dataset does not provide explicit force measurements in Newtons, as Pokeflex does. The PLUSH dataset reports the pose of the nozzle that applies an air stream to the different Plush toys and it provides a table (https://github.com/facebookresearch/plush_dataset/blob/main/plush_data.pdf) that reports the impact of the nozzle wrt to the distance on a weight. However, the Plush dataset does not report any method to convert such quantities to forces and they do not evaluate the accuracy of the force estimation. To clarify this, we added a new footnote “by providing air nozzle poses” to Table 2 (L117-118) in the latest version of the paper .

  Furthermore, we want to emphasize that PokeFlex is not proposed as a simple point cloud to mesh reconstruction dataset. PokeFlex provides real-world high-quality soft-body temporal deformation data using both high-end and consumer-grade capture systems, synchronized with robot interaction sensor data. Our ultimate goal is to investigate the modeling of the physical properties of soft-body objects in the context of manipulation tasks. Accurate modeling of physical properties is not possible with most datasets, as they lack annotations of the exact point of force exertion and the applied force.

  As the reviewer notes, the Plush dataset is an important step towards this goal. However, in the Plush dataset, the forces applied on the objects are inevitably more global using an air stream, which causes more global deformations and hinders accurate force estimation. Such global deformation can lead to high inaccuracies, as they report in their paper:

  "Larger errors are observed for objects with a thin and tall component (see the last 4 rows of the table). This error is largely caused by tracking inaccuracies of the nozzle: even slight inaccuracies can cause large errors when, for example, the neck of the dinosaur moves while the recorded air stream direction does not, or barely, touch the objects". 
  

  Regarding the second type of manipulation in the PLUSH dataset, using fishing lines, the dataset does not report tension forces of the fishing lines as the authors only incorporated such forces as soft position constraints for specific points.

  Finally, we remark that Pokeflex applies local deformations at specific contact points and also deals with rigid surface contacts. While we acknowledge that the Plush dataset itself is very valuable, PokeFlex incorporates significant complementary information.

• W1 - Importance of mesh representation and potential applications: See general response 2.

• W2 - Generalizability of networks on unseen objects: The main contribution of the paper is the dataset. The presented online 3D mesh reconstruction is just one of the many potential downstream applications. Nonetheless, we agree that this is an interesting experiment. As such, we include the results in the latest version of the paper (see Appendix A.7).

  • Appendix A.7: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA7.pdf

  For the generalizability experiments, we trained reconstruction models using sequences of synthetic point clouds or images as input. We trained on 13 objects and evaluated on 4 unseen objects, namely, the foam cylinder, the plush volleyball, the sponge, and the toilet paper roll. As a reference, Table 10 and Table 11 in Appendix A.7 also provide the performance on the validation set of 13 objects that were used during training.

  We note that the model trained on point clouds tends to generalize better than the model that uses images. As expected, given the size of our dataset, the performance degrades for unseen objects. However, some level of generalization can be observed for objects like the plush volleyball that has a similar shape and size as other objects in the training set.

• W3 - Technical novelty of the network and ablation studies: We agree that the technical novelty of the model is limited. However, we remark again that the main focus of the paper is the dataset. Regarding the ablations, we consider that the evaluations of different data modalities and their combinations for 3D mesh reconstruction serve as ablations, as they highlight the different performance results that can be obtained.

We hope that you found our new experiments and additional information valuable. As the end of the rebuttal is approaching, we would like to ask if you have any remaining concerns. If not, please consider raising your score accordingly.

• W1 - Other types of deformable objects and further applications: We agree with the reviewer that cloth items and cables are very important. However, as mentioned in L521-522 , the focus of PokeFlex is on volumetric objects, which we argue is also an important category of deformable objects and can be used as a testbed for multiple downstream applications. See General responses 1 and 4.

• W2/W3 - Action protocols. See general response 4.

• W4 - Related work:

  • [1] Regarding DOFS we remark that their work was presented as a workshop paper at CoRL 2024, which took place one month after the deadline submission for ICLR. It is not common practice for the authors of PokeFlex to cite workshop papers. However, we acknowledge that their work is relevant and it is concurrent to ours. DOFS is a pilot dataset that contains only one type of deformable object under quasi-static deformation, few camera views, and it does not report interaction forces. In contrast, PokeFlex includes multiple objects, with annotated time sequences of multiple data modalities that include interaction forces and contact points. For completeness, the latest version of our paper now includes DOFS in the related work.

  • [2] CEPB primarily focuses on synthetically generated photorealistic images in static configurations, which fundamentally differ from the focus of PokeFlex. While CEPB demonstrates the utility of simulation for generating large-scale datasets, it does not include real-world dynamics of deformable objects or forces as presented in the PokeFlex dataset.

• Q1 - Dealing with fine-grained details: As stated in L459-460 L431 “Better fine-grained reconstruction results can be expected by rearranging the cameras in a smaller workspace”. For a smaller workspace, a smaller number of cameras should be enough to generate 360° reconstructions, which would in turn reduce the compute time required to generate the ground truth meshes. In summary, the MVS system can enable future iterations of the dataset, focusing on fine-grained reconstruction of smaller deformable objects.

• Q2 - Plans to incorporate cloth items: Yes, The current MVS setup already enables the reconstruction of cloth objects. (See Video 1). Future iterations/extensions of the dataset will include cloth-like objects. However, as the methods used for the reconstruction and manipulation of deformable objects are quite diverse depending on the type of deformable object (volumetric, thin-shell (cloth), linear), we decided to focus the scope of PokeFlex on volumetric deformable objects initially.

  • Video 1: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/videos/cloth_reconstruction_example.mp4

• Q3 - Other action protocols: See general response 4.

• Q4 - Additional annotations: Yes, we will provide a script to generate segmentation masks, which can be easily obtained using the 3D textured meshes that we provide.

Note 1: For clarifications about the exclusion of the Pizza object see response to reviewer gn5G. A new custom made Pizza will be introduced later.

Note 2: Due to compute limitations, Table 4 does not cover some of the modalities presented during the initial submission, but it will be completed during the rebuttal period, with experiments trained on all objects for Kinect point clouds, sparse synthetic point clouds, and a new combination of point clouds with force information.

Specific concerns

Further clarifications are provided in the responses to each reviewer, including the positioning of our work wrt concurrent related papers that became available on Arxiv several weeks after the ICLR submission deadline, as suggested by m6Yb, BD6s.