BricksRL: A Platform for Democratizing Robotics and Reinforcement Learning Research and Education with LEGO

Response Weaknesses:

Thank you for your assessment of the weaknesses, we are happy to address these points to provide more clarity.

We recognize the potential benefits of increased communication speed and are willing to improve this. We've contacted PyBricks directly to collaborate on enhancing communication rates and they claim to have better protocols almost ready. Once implemented they will be made available in BricksRL immediately. However, we hardly had any problems due to the slower communication speed. As described in the paper, lower communication even led to more stable and faster learning for the walker robot.

The policies learnt were robust notwithstanding existing backlashes and noise. For the real-world training, we did not take any measures to stabilize the learning as it was not necessary. However, as mentioned in the paper, for the sim2real transfer with the policies trained in simulations we added noise to the actions to simulate backlashes and sensor noises. This was sufficient to learn stable policies that could be evaluated successfully in the real world.

Response Questions:

• The robots were mainly trained online in the real world. We only developed two simulations to demonstrate the sim2real capabilities. Our approach to simulating LEGO robot models in BricksRL is straightforward but effective. In the two simulation environments, we simulate the transition dynamics by directly applying the actions to the simulated motor angles. This works well because our action space represents the delta angles for robot movement, allowing for a simple yet accurate simulation of the robot's behavior.
• Yes, the RoboArm-mixed-v0 is fully trained in the real world, we did not develop any simulator for this environment. The fact that LEGO can be used in this setting proves the value of the LEGO platform for RL research.
• Yes, the trained policies are robust to different sensor noises and backlashes. Different sensor noises/deviations are more harmful than others. For example, we did notice that during training of the RoboArm-mixed-v0 with image inputs lightning conditions are very important and need to be stable during the whole training. However, this is more specifically for the contour detection of the ball for reward calculation and not so much for policy learning.

Response Weaknesses:

We greatly appreciate your valuable input and suggestions for improving the clarity of our paper. We are pleased to incorporate the proposals as far as possible in the final version of our paper.

Response Questions:

We're happy to provide more details about our LEGO robot setups:

Robot longevity and motor wear:

• Battery life varies depending on the robot's complexity and usage intensity. For example, our walker and RoboArm models typically operate for 3-4 hours on a single charge, but it is possible to directly provide power via a USB connection to a larger battery pack.
• Static robots like the RoboArm can be directly connected to a power supply via a USB for extended operation in training and execution
• Throughout all our experiments and tests over a couple of years, we haven't experienced any part failure, indicating good durability. If replacement is needed, individual motors cost between 40-60€.

Wired connections for training:

• This is possible for stationary robots using the USB connection.
• We recognize the potential benefits of increased communication speed. PyBricks is already experimenting with better communication channels and they will be made available in BricksRL immediately.

Cost of setups:

• Our setups utilize components that can be largely found in the LEGO Education kit and extension kit. The current price range for these sets is approximately 700€.

Response Limitations:

The main limitations of LEGO components are in the tolerance of the gears, precisions of the encoders, and torque of the motors. However, in our experiments, we have shown that BricksRL is capable of working around those. Our extensive testing and experiments with the LEGO robots have yielded very robust results.

We further show the robustness in the sim2real experiments, where policies trained in idealized simulations were directly transferable to real robots with minimal issues. Our approach of adding small amounts of noise to simulated observations to mimic backlash and sensor inaccuracies proved effective in bridging the reality gap.

Regarding the limitation of communication speed, as mentioned above Pybricks is working on better communication channels, and once available those will be immediately included in BricksRL.

Response Weaknesses:

• Our primary contribution lies in providing accessible, low-cost hardware integration for real-world robotics experimentation. Our goal is to lower the barrier to entry for robotics research and education, enabling a wider audience to engage with physical robots beyond simulations. LEGO parts are widely available, robust, and reproducible because they are mass-produced. We are not aware of any other robotic platform with such characteristics, making this paper a first instance of that.

• The demonstrated tasks serve as proof-of-concept examples to validate the framework's functionality. It is not at all obvious that LEGO can be used for RL tasks, as it is not designed for that. Gears have generous tolerance, the stepping motors offer encoders but the precision is about 1 degree, and structurally, they are made of bricks. In this paper, we show that all this hardware can be used effectively for RL.

• Our focus is on demonstrating the framework's compatibility with a range of algorithms rather than pushing the boundaries of RL performance given that this is a totally new platform. The outcome is a validated platform for the democratization of robotics. However, we made sure to implement the RL framework using the official RL library from PyTorch. This allows direct usage of all state-of-the-art RL algorithms. Since submission, we have already expanded our example experiments to include several offline RL algorithms, such as BC, TD3+BC, IQL, and CQL, trained on collected datasets. We've also integrated pre-trained foundation models like VIP transform from TorchRL. Which can be used to give reward signals to sparse reward tasks but can also be utilized to label hand-collected datasets that have no reward signal. VIP can enable simple, few-shot offline RL on real-world robotic tasks for low data quantities. We are making available these examples and the offline datasets with the revised version of the paper.

• The high variance in the plots is potentially stemming from the noisy nature of real-world robotics. This variability itself is an interesting area for future research, investigating robust learning methods in noisy, real-world environments.

Response Questions:

We apologize for the lack of details on policy architectures in the current version of the paper. We will add this information in the appendix. All examples are available in full detail in the GitHub repository. Our default architecture for both policies and Q-functions is a multilayer perceptron (MLP) with three linear layers and ReLU activation functions. We also incorporate dropout and layer normalization specifically for the DroQ algorithm.

• For the vision-based environment (roboarm-mixed-v0), we employ a hybrid architecture. This combines image observations with state observations of the robot angles. Specifically, image observations are processed through a convolutional neural network (CNN), while state observations are encoded using an MLP. The resulting image and state embeddings are concatenated and passed through a final output head. We have added a detailed description of this architecture in the appendix of the paper.

• Regarding the concern about increased training time and potential stress on LEGO hardware. We have run the same LEGO components for almost two years to prepare this paper with zero faulty parts. LEGO parts are created for kids and mass-produced so they don't break.

Response Limitations:

Thank you for raising these important points about the limitations of low-cost robots. The main limitations of LEGO components are in the tolerance of the gears, precisions of the encoders, and torque of the motors. However, in our experiments, we have shown that BricksRL is capable of working around those. Our extensive testing and experiments with the LEGO robots have yielded very robust results. Throughout our trials, we did not encounter a single failure of any motor or gear, which speaks to the reliability of these components even under repeated use. Regarding backlash and sensor noise, these factors are certainly present in our setup. However, their impact on policy learning and execution has not been significant. This is evidenced by our successful sim2real experiments, where policies trained in idealized simulations were directly transferable to real robots with minimal issues. Our approach of adding small amounts of noise to simulated observations to mimic backlash and sensor inaccuracies proved effective in bridging the reality gap. The architecture ensures that the computational load of training and inference remains separate from the physical robot, mitigating concerns about hardware reliability or potential breakages due to increased model complexity. The LEGO components are primarily involved in the physical execution of actions and collection of observations, tasks which are not affected by increases in model size. We have added this information to the main manuscript.

Response Weaknesses:

Thank you for your thoughtful feedback, we appreciate your recognition of its practical utility and educational value. However, we respectfully disagree with the assessment that the scientific contribution is limited.

We would like to emphasize several key points that demonstrate the significant scientific value of BricksRL:

• BricksRL is the first platform to seamlessly integrate state-of-the-art reinforcement learning algorithms with hardware that is economical, modular, mass-produced, and easy to use like LEGO. This allows everybody in the world to focus on RL algorithms but test their ideas on real robots straightaway. There are over 1M LEGO hubs sold worldwide which are compatible with Pybricks and BricksRL as of today. We cannot emphasize enough how transformative this aspect is in the field of AI for robotics.
• By enabling the creation of low-cost, customizable robots, BricksRL addresses a critical challenge in RL research - the ability to conduct reproducible experiments on multiple real robots at once. This scalability and reproducibility is crucial for advancing RL algorithms in real-world settings.
• We have shown with BricksRL that complex RL algorithms like SAC, TD3, and DroQ can be successfully applied to train LEGO robots end-to-end in the real world within reasonable time frames. This is a necessary validation and demonstration of the platform towards making real-world RL more accessible and practical.
• We have also demonstrated the ability to incorporate non-LEGO sensors to increase its potential for broader applications in robotics research, beyond just LEGO components. LEGO itself might provide more sensors in the future.
• By providing built plans and training results, BricksRL contributes to the reproducibility of RL experiments in robotics, a crucial aspect of scientific research often challenging in real-world settings. BricksRL has clear educational benefits, but we have also created it for its research potential. We believe that BricksRL's contribution represents a significant step towards democratizing real-world RL research in robotics, addressing key challenges in the field such as cost, scalability, and the reality gap.

Offline datasets:

We have already generated valuable offline datasets with our LEGO robots. As part of our revised paper submission, we are making these datasets publicly available in the code repository to facilitate research in offline RL that is directly available in TorchRL.

We have curated datasets for the three robot configurations: 2Wheeler, Walker, and RoboArm. These datasets encompass both expert and random data, specifically tailored to the tasks explored in our experiments.

For the 2Wheeler, we offer datasets for the 'RunAway-v0' and 'Spinning-v0' environment tasks. The Walker robot dataset focuses on the 'Walker-v0' task, while the RoboArm robot dataset addresses the 'RoboArm-v0' task. The expert data for each dataset was carefully collected using the following process: We first trained a Soft Actor-Critic (SAC) agent to successfully complete the specific task. Upon achieving satisfactory performance, we evaluated the agent over 100 episodes per task, recording all transitions. For example, the evaluation process took approximately one hour for the Walker robot and yielded 10,000 expert transitions.

To complement the expert data, we also generated random datasets. These were created by executing a random policy in the environment, similarly for 100 episodes per task. This process resulted in eight distinct datasets:

1. RunAway-v0-expert
2. RunAway-v0-random
3. Spinning-v0-expert
4. Spinning-v0-random
5. Walker-v0-expert
6. Walker-v0-random
7. RoboArm-v0-expert
8. RoboArm-v0-random

We trained successfully offline-RL algorithms, including BC, TD3+BC, IQL, and CQL using these datasets. This allows researchers to compare the performance of various RL algorithms, both offline and online, when trained on expert demonstrations versus random interactions. Such comparisons can provide valuable insights into the effectiveness and adaptability of different RL methods across various data quality levels and robot configurations.

Response Questions:

Currently, our approach to simulating LEGO robot models in BricksRL is straightforward but effective. In our experiments, we simulated the transition dynamics by directly applying the actions to the simulated motor angles. This works well because our action space represents the delta angles for robot movement, allowing for a simple yet accurate simulation of the robot's behavior. BricksRL does not yet support URDF model construction, which would allow for more sophisticated simulations and easier integration with mainstream simulators like MuJoCo or Isaac Sim. However, a few LEGO CAD (https://www.leocad.org/) are actually available. One could build completely virtual robots using only LEGO parts. The complete list of parts is then listed. The LEGO ecosystem is maybe unsurprisingly very rich. It would be therefore time-consuming but relatively straightforward to build simulated LEGO robotics. We are considering this avenue of developing MuJoCo-based LEGO simulations for the future. This is, however, secondary, to prove that RL research for robotics can be done at all with LEGO which is the outcome of this work.