FruitBin: A tunable large-scale dataset for advancing 6D Pose estimation in fruit bin picking automation

Dear Reviewer,

We are grateful for your comprehensive review and constructive feedback on our manuscript. We have taken the time to address each of your comments and concerns as follows:

Photorealistic Rendering:

We concur with your observation regarding the limitations of Gazebo in terms of photorealistic rendering. While state-of-the-art generators such as Blenderproc and Kubric offer some advantages, they lack crucial occlusion annotation. Furthermore, we emphasize that models trained on our dataset can be integrated and tested for robotic simulation, a feature not guaranteed with Kubric and Blenderproc (we are not certain that the PBR rendering can be integrated on a camera in Pybullet, for example).

We are cognizant of these limitations and believe that domain randomization is sufficient to reduce the sim2real gap. Moreover, as a general way of addressing the recurrent sim2real gap, data augmentation or domain adaptation [4] can further reduce the sim2real gap, and we demonstrated that a simple diffusion model can generate a good variety of realistic backgrounds.

Variation in Fruit Size and Shape:

While our current focus is on 6D pose estimation, we acknowledge the lack of variation in size and shape. Although not part of the benchmark yet, we leveraged PickSim to generate testing data where random scale variation over the three directions is applied to vary the shapes. Additionally, to answer real-world scenarios of industry and supermarkets, we generated bin picking for only one type of object. This comes with two variations: one where the mesh is unique and the second where mesh modification is applied.

Real Test Set:

We agree with your assertion on the importance of a real test set for sim2real and real-world applications. To this end, we intend to augment our dataset with real-world data to create a test set for evaluating sim2real performance.

Benchmarking Methods:

We concur that our benchmarks should include more recent methods. Although Pvnet and Densefusion are getting outperformed by recent methods, we believe that they remain representative baselines. It is worth noting that even with the recent result of the 6D pose BOP challenge 2023 [3], the difficulties are still the same. We can highlight non-negligible performance differences for occluded or cluttered datasets (LM-O, IC-BIN, ITODD) with scores below 0.8 and better than 0.9 for less occluded and cluttered ones ( T-LESS, YCB-V). This strengthens the need for occlusion studies in 6D pose estimation and benchmarks.

We understand the importance of strong baselines in demonstrating the effectiveness of our approach, and we are working to integrate GRD-NET[1][2], which has been “The Best Open-Source Method” for the benchmarks in the BOP challenge in 2022 and 2023[3] to provide a more precise evaluation of our dataset.

Robotic Applications:

We appreciate your suggestion to demonstrate robotic applications such as bin picking using our dataset. For this rebuttal, we haven’t yet included grasping with the real robot. However, as an early proof of concept, a grasping simulation pipeline has been set up as follows:

• we automatically generate a grasping list associated with the mesh,
• we evaluate the 6D pose with our trained Densefusion model
• we used free obstacle path planning libraries such MoveIt to grasp the object with the list of grasp and the 6D pose of the object

Symmetric Objects in Table 2: We thank you for your suggestion and denoted symmetric objects with an asterisk (*) in Table 2 in the revised version of the paper.

Writing Issues: We will address the writing issues you pointed out, including reducing the size of Table 1, separating additional work from the main paper, and improving the marking of symmetric objects in Table 2.

Questions:

PVNet and GT Bboxes: In all our generated benchmarks, we have only considered 1 instance of a category in the data. The filtering parameters for the benchmarks"

Diffusion-Generated Backgrounds: The diffusion backgrounds have only been tested to showcase the feasibility of improving the sim2real gap. However, we haven’t explored how it would impact the model for 6D pose estimation. We hope that our responses address your concerns.

[1]Wang, Gu et al. “GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 16606-16616.

[2] https://github.com/shanice-l/gdrnpp_bop2022

[3] https://bop.felk.cvut.cz/challenges/bop-challenge-2023/

[4] Truong, J., Chernova, S., & Batra, D. (2021). Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. IEEE Robotics and Automation Letters, 6(2), 2634-2641.

Dear Reviewer,

Thank you for your review and positive feedback on our paper. We are pleased that you found our research direction and dataset valuable for the industry.

We appreciate your suggestion regarding the need for more robust baselines. Although Pvnet and Densefusion are getting outperformed by recent methods, we believe that they remain representative baselines. It is worth noting that even with the recent results of the 6D pose BOP challenge 2023 [3], the difficulties are still the same. We can highlight non-negligible performance differences for occluded or cluttered datasets (LM-O, IC-BIN, ITODD) with scores below 0.8 and better than 0.9 for less occluded and cluttered ones (T-LESS, YCB-V). This strengthens the need for occlusion study in 6D pose estimation and benchmarks.

We understand the importance of strong baselines in demonstrating the effectiveness of our approach. We are working to integrate GRD-NET[1][2], which has been “The Best Open-Source Method” for the benchmarks in the BOP challenge in 2022 and 2023[3], to provide a more precise evaluation of our dataset. Once again, we thank you for your time and constructive feedback. We look forward to incorporating your suggestions to improve our work.

[1]Wang, Gu et al. “GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 16606-16616.

[2] https://github.com/shanice-l/gdrnpp_bop2022

[3] https://bop.felk.cvut.cz/challenges/bop-challenge-2023/

Dear Reviewer,

We are grateful for the time and effort you have invested in reviewing our paper. We appreciate your feedback and would like to address your concerns.

Limited Contribution:

We acknowledge your concern regarding the technical contributions of our paper. However, we believe that our datasets, specifically designed for 6D pose estimation in bin picking, could significantly advance the development of 6D pose estimation models. Our unique dataset brings together major challenges in 6D pose estimation, and its large scale allows for the creation of specific benchmarks. This is not proposed by the current existing datasets. The recent 2023 BOP challenge results highlight that state-of-the-art 6D pose estimation models are still sensitive to complex scenes and occlusion, as evidenced by a drop of 0.1/0.2 for datasets with occlusion and bin picking, indicating room for improvement.

Regarding the lack of real-world data, we agree that this is a significant limitation. We are currently collecting data from physical setups to enhance our dataset, allowing for a better evaluation of the sim2real gap and the performance of models under real-world conditions.

Inconvenient Platform: We selected ROS+Gazebo due to its widespread adoption in the robotics community and its compatibility with various hardware. While Mujoco, PyBullet, and Isaac Gym are gaining popularity in the robot learning community, we maintain that ROS+Gazebo remains one of the most utilized platforms for robotic simulation [1]. To our knowledge, Mujoco, Pybullet, and Isaac Gym may be more low-level robot learning-oriented than Gazebo+ROS. The latter offers the advantages of a large community and integrates popular high-level libraries such as MoveIt. The default use of ROS enables users to easily apply their developed pipeline to real robots. Specifically to robot learning, even if maybe less popular, Gazebo is still a reliable choice [2,3].

Format Issues: We apologize for the formatting issues in Table 1 and the references. These will be corrected in the revised version of the paper, along with the supplementary materials detached from the main paper.

Questions: In response to your query, we do indeed plan to include samples collected in the real world in the dataset. We believe this enhancement will increase the dataset’s value for the community.

[1] Collins, J., Chand, S., Vanderkop, A., & Howard, D. (2021). A review of physics simulators for robotic applications. IEEE Access, 9, 51416-51431.

[2] Zamora, Iker, et al. "Extending the openai gym for robotics: a toolkit for reinforcement learning using ros and gazebo." arXiv preprint arXiv:1608.05742 (2016).

[3] Ferigo, D., Traversaro, S., Metta, G., & Pucci, D. (2020, January). Gym-ignition: Reproducible robotic simulations for reinforcement learning. In 2020 IEEE/SICE International Symposium on System Integration (SII) (pp. 885-890). IEEE.

Question:

In response to your query about modeling haptic feedback for delicate fruits, we currently do not incorporate this aspect into our dataset, as our primary focus is on pose estimation. We have clarified this point in the paper.

Regarding object diversity, other benchmark datasets typically include between 8 (occluded linemod) and 39 objects (excluding GraspNet-1B, which primarily focuses on 6DOF grasping rather than 6D pose estimation). This is generally more than our dataset, as we focus solely on fruits. However, it’s important to note that even if the number of objects is low, datasets can pose a significant challenge for the 6D pose estimation community, particularly with occlusion. For instance, the popular dataset “occluded-Linemod” also has only 8 objects but remains one of the most challenging datasets. Indeed, in the recent BOP challenge 2023 [4], this dataset was one of the most difficult, with a score of 0.794 achieved by the winner of the challenge. This is compared to 0.928 for “YCB-video” which has 21 objects, or even “HomebrewedDB” with a score of 0.950 with 33 objects.

We hope this addresses your concerns.

[1] Zhongkui Wang, Shinichi Hirai, and Sadao Kawamura. Challenges and Opportunities in Robotic Food Handling: A Review, jan 2022. ISSN 22969144.

[2] Tobin, Josh, et al. "Domain randomization for transferring deep neural networks from simulation to the real world." 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017.

[3] Truong, J., Chernova, S., & Batra, D. (2021). Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. IEEE Robotics and Automation Letters, 6(2), 2634-2641.

[4] https://bop.felk.cvut.cz/challenges/bop-challenge-2023/

Dear Reviewer,

Thank you for your insightful comments and constructive feedback. We appreciate the time and effort you’ve put into reviewing our paper. We agree with your concerns and have addressed them as follows:

Real-world applicability: We acknowledge the limitations of synthetic datasets in replicating real-world conditions. Precise evaluation of our dataset for real fruits is an important question, given that we only have synthetic data. We are currently scanning 3D real fruits and adding 6D pose real-world data to extend our dataset. This dataset will be used for evaluating the sim2real gap of our dataset.

Variety of objects: We agree that by targeting fruit bin picking, we inevitably limit the variety of shapes. Fruits are indeed mostly smooth compared to artificially created objects, such as industrial objects. However, fruit bin picking is a delicate task that requires careful handling to avoid damaging the fruits. We believe that knowing the semantics of the fruit and its position in the scene is of major importance during the grasping process to avoid any damage.

Necessity of 6DOF pose estimation: We believe that 6DOF pose estimation isn’t incompatible with simple objects, and it can provide more precise control for robotic arms, even when using suction grippers. However, suction grasping requires fruits to be smooth, like apples, pears, or maybe bananas. In practical use, it will not be suitable for rugous textures such as lemons, oranges, or kiwis.

Intended use cases: The dataset’s intended uses are multiple, with a goal of making it general for multiple purposes:

• The first intended use is benchmark making as it gathers major challenges in 6D pose. It comes as a challenge for the community to make improvements in 6D pose estimation as it is commonly used with the popular BOP challenge.
• In addition to these existing datasets for benchmarking, ours offers a useful and practical scenario that could easily occur in daily scenarios: fruit industry, house fruit bin, or even supermarket.
• Mixing fruits in a bin is the most general and difficult scene we could create for 6D pose estimation purposes. However, it is right to note that real-world scenarios, such as industry or supermarkets, usually deal with only one category. In order to address this, we extended our dataset with data with bins of only one category of object.

Diversity of the dataset: We understand your concern about the diversity of the dataset. We have included 8 fruits that we believe are the most common (apple, apricot, banana, kiwi, lemon, orange, peach, and pear). However, it remains in the range of the number of objects used by 6D pose datasets.

Lighting variations: We agree that real-world visual phenomena like shadows and reflections are important. We are cognizant of these limitations and believe that domain randomization is sufficient to reduce the sim2real gap. Moreover, as a general way of addressing the recurrent sim2real gap, data augmentation or domain adaptation can further reduce the sim2real gap, and we demonstrated that a simple diffusion model could generate a good variety of realistic backgrounds and can create shadows or reflections.

Language choices: We have revisited the language by removing some unnecessary adjectives.

Comparison with robotic gripper: We appreciate your suggestion to demonstrate robotic applications such as bin picking using our dataset. For this rebuttal, we haven’t yet included grasping with the real robot. However, as an early proof of concept, a grasping simulation pipeline has been set up as follows:

• We automatically generate a grasping list associated with the mesh.
• We evaluate the 6D pose with our trained Densefusion model.
• We used free obstacle path planning libraries such as MoveIt to grasp the object with the list of grasp and the 6D pose of the object.

Paper :

We have corrected numerous typos throughout the paper. Moreover, we have streamlined Table 1 by eliminating the column that pertains to image resolution. We have also marked symmetric objects in Table 2 with the symbol ‘’. We have enhanced and clarified sections of the paper, particularly those discussing adding supplementary data, the necessity of 6D pose estimation for fruit bin picking and the application of real robots. We also mention the creation of a real-world dataset. The primary changes have been highlighted in red in the revised version of the paper.