DittoGym: Learning to Control Soft Shape-Shifting Robots

Why are these tasks selected? What exactly is it that they evaluate? Why is it important to have them instead of others? Why the proposed tasks were selected, and clarify the experimental setup. In addition, it would be great to include more relevant benchmarks for this problem.

Our task selection follows both generic RL benchmark design principles and the specific needs of reconfigurable robots. We highlight the factors we hoped to cover in our benchmark and the corresponding factor vector of each environment:

• Tasks have to cover a range of difficulties (A=easy, B=hard)

• Some tasks evaluates robot itself, others requires the robot’s ability to interact with external objects (A=not require, B=require)

• Some tasks require just one morphology change, while others need changes of morphology multiple times (A=one change, B=multiple changes)

• Some have non-convex reward landscape that requires algorithm to be long-horizon (A=short-horizon, B=long-horizon)

• Some features softer material and some features less soft material (A=softer, B=less softer)

  Task	Factor Vector
  SHAPE_MATCH	[A, A, A, A, A]
  RUN	[A, A, A, A, B]
  GROW	[A, A, A, A, A]
  KICK	[B, B, A, A, B]
  DIG	[B, B, A, A, B]
  OBSTACLE	[B, A, B, B, B]
  CATCH	[B, B, B, B, A]
  SLOT	[B, B, B, B, B]

Each task is designed to represent certain factor combinations that we deem important, and we believe they together can give users a comprehensive understanding that a lot of RL ablations do through choices of tasks.

We agree that more benchmarks would be helpful! However, our problem is the first of its kind. Reconfigurable robot benchmarks are not possible before us due to the lack of formalization and difficulty to make physical simulations, which is our main contribution. While we only have 8 environments in our paper, these two contributions make it much easier for the RL community to contribute more reconfigurable robot tasks. We plan to gradually add more tasks to the benchmark to achieve higher impact in the long term but we also believe that quality of benchmark is more significant in a first benchmark.

I am not sure I understand how the baseline of Neural Field Policy is implemented and whether this is a novel contribution of the paper or some other work proposed before for this problem.

Sorry for the confusion. Neural Field Policy is designed to be a method that can take advantage of our infinitely dimensional action space by querying values in a coordinate-conditioned way. We recognize the confusion it caused, and will replace it in the camera-ready version with the two added baselines following your suggestion.

I don't think Fig. 6 is helpful, the concept it represents is trivial and easily explained in a sentence

Thank you for your suggestion! While the concept is simple, Fig. 6 is also designed to be a visualization of the resulting shape of different policies, just like how RL paper also demonstrates suboptimal runs of baselines in their videos to give viewers more insights. We will add more visualizations of different tasks in the camera-ready version.

---

[26] Sun et al. (2022). Reconfigurable magnetic slime robot: deformation, adaptability, and multifunction. Advanced Functional Materials, 32(26), 2112508.

[27] Fu et al. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.

[28] Li et al. (2022). Hierarchical planning through goal-conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 7(4), 10216-10223.

[29] Nachum et al. (2019). Why does hierarchy (sometimes) work so well in reinforcement learning?. arXiv preprint arXiv:1909.10618.

[30] Pathak et al. (2019). Learning to control self-assembling morphologies: a study of generalization via modularity. Advances in Neural Information Processing Systems, 32.

Thank you for your helpful comments. We respond to your comments below as well as adding more baselines.

It is unclear whether the proposed simulator and benchmark is a good model of reality. Is it modeling a specific robot or class of robots? Could there be some experiments to show the validity of the simulator? I'm not necessarily referring to simulation to reality transfer, but rather to show that the simulator dynamics are correlated with the dynamics of a real robot.

Thank you for pointing out the real world connection! We took great caution throughout the project to make sure everything is governed by real-world physical equations. Our physical simulation is based on MPM, a widely used simulation algorithm that’s designed to simulate real world materials like liquids and plastics. Our robot material is simulated in a way governed by equations of the real stress-strain curve of plastic-elasto materials. The actuation is governed by that of magnetic field (which is used to control the first real-world highly reconfigurable robot [26]), and follows the original implementation in Taichi that simulates point gravity attraction. The only change we made is to normalize the actuation to follow conservation of momentum because we are parameterizing internal forces only. This is done before feeding actuation into simulation and doesn’t break any physical equation itself.

The proposed hierarchical reinforcement learning approach is relatively standard. Its application to the problem of morphology-changing robots might be new, though.

Thanks for mentioning HRL, a nice potential addition to our related work. While our method has a spatial resolution hierarchy, it’s quite different from the hierarchy in HRL. In HRL, high-level policy and low-level policies are designed to be disentangled, such that they never share the input-output behavior. Rather, the high-level policy’s output is usually fed to low-level policy as input. For example, The famous D4RL [27] environment “ant maze” is widely used in HRL [28][29], where a high-level policy outputs local goals and low-level policy outputs actuations.

In contrast, our policy of different resolutions shares the same 2D input and output space. The action field of high-resolution policy and that of low-resolution are directly added algebraically (The weighting mask and feed of coarse action into fine policy are optional). This is because of our hierarchy in the sense of spatial hierarchy, like CNNs and Feature Pyramids in computer vision, rather than the hierarchy in HRL. We appreciate the relation you pointed out and would add explanations to our paper upon acceptance.

I don't understand the reasoning behind the proposed baselines. There seem to be other approaches that are more related to the problem (e.g. Pathak et al., Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity; Whitman et al., Learning Modular Robot Control Policies). Having a more detailed comparison could give a better idea of how difficult the proposed benchmark for current algorithms is.

Thanks for bringing up these baselines. We’d like to clarify that suggested baselines aren’t directly applicable here while also providing results of the adopted version of suggested baseline [30].

• Clarification: highly-reconfigurable robots lack articulated joints, links and thus lack of explicit topological structure these baselines need, as we highlighted in general response.

• Additional experiments: Despite that, we adopt the suggested Modular Policy baseline[28] by assigning topology with k-means clustering so GNN can be applied. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our rebuttal website. Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

While our method has a spatial resolution hierarchy, it’s quite different from the hierarchy in HRL. In HRL, high-level policy and low-level policies are designed to be disentangled, such that they never share the input-output behavior

Details of both baselines and result curves are added to our rebuttal website. Figure 1.1 in the website illustrates the cluster while Figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Dear Reviewer UEFW,

We first thank you again for your comments and suggestions.

In our earlier response, we have provided detailed clarification based on your questions about the reality meanings of our proposed method and the settings of our benchmark. In addition, we illustrated the unique technique contributions of our method. Furthermore, we have conducted additional experiments with strong baselines to illustrate the effectiveness of our proposed method on our rebuttal website following your great suggestion. Finally, we have provided detailed analysis about the characteristics of tasks in our benchmark to illustrate the rationality of the task designs.

As we are ending the stage of the author-reviewer discussion soon, we kindly ask you to review our revised paper and our response and reconsider the scores if our response has addressed your concerns.

If you have any other questions, we are also pleased to respond. We sincerely look forward to your response.

Best wishes!

The authors

Thank you for raising the score!

why the term hierarchical was used

Sorry for the confusion. While hierarchical is widely used in HRL, it's also a term in computer vision used widely in super-resolution and feature pyramid (anything that has multi-resolution stage or representations). We will try to avoid such conflict of terms in camera-ready.

Thank you for your helpful comments. We respond to your comments below.

A potential weakness of the approach, especially when it should ultimately be transferred to real robots, is that the control of the robot's shape does not happen based on local interactions, i.e. the employed controller has to take into account the complete shape of the robot and its environment.

Thank you for making the connection to real robots. First-person-view observation and third-person view observation are both common choices in real robot reinforcement learning papers. We focus on third-person-view exactly because the current real-world hardware implementations [20] [21] of the robot uses a third-person view setup. While we recognize that reconfigurable robot research may evolve in the long-run, we believe our setup and benchmark is best suited for current hardware.

Missing relevant literature on evolving soft robots and simulators (e.g.VoxCad). "Cheney, Nick, et al. "Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding." ACM SIGEVOlution 7.1 (2014): 11-23."

Thank you for the mention and we will add it to related work! However, we'd still like to emphasize that our problem is very different from co-design, as we explained in the general response.

However, these search-based zeroth-order optimization methods are computationally demanding and inefficient” - is this shown in the paper? It's a common argument but it would be good to have some references here backing it up.

Thanks for your suggestion. We will add more references to support this claim in camera-ready.

Importantly, what are the lessons for the larger machine learning community? e.g. for which other tasks could the coarse-to-fine (CFP) algorithm be useful?

Coarse-to-fine has seen its success in many machine learning domains like image and video generation [22][23]. However, it’s been largely ignored by the RL community because not many RL problems largely ignore the structures of action space. The lesson is that whenever resolutional hierarchy exists, don’t overlook it just because this is not traditionally used in your domain.

Since this conference is about representation learning, it would be interesting to investigate further what type of representations the RL algorithms learn to control these soft robots. How do the coarse and fine-grained control interact to solve the tasks at hand?

Thank you for defending the relevance of the conference. From the “About Us” page of ICLR, “The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning.”

Therefore, as a deep reinforcement learning paper we naturally fit into the focus of ICLR.

Is the code available?

Yes. We plan to release the code for both benchmark and algorithm upon acceptance.

How computationally expensive is the simulation environment?

We made it really fast! The simulation is GPU based. We report the usage and speed of our most computation-intensive task DIG (because it contains the most particle numbers). Simulation alone, the environment runs at an average speed of 70.1 FPS on a RTX4090 GPU, with only 20-30% volatile utility usage and 15% memory usage. Therefore theoretically you can run 5 of such simulations on RTX4090 at similar speed.

How large is the action space?

Although our formalization allows an infinite dimensional action space, our finest policy runs with an output of size 21616 = 512 dims, which gets interpolated to this infinite dimensional action space.

How is the upsampling of coarse actions done? What are the network details (e.g. number of layers etc.?)

Sorry about the confusion. The upsampling of coarse action is done by the bicubic interpolation while the network details are shown below:

• Critic Network: The 3-layer CNN based encoder can embed the input of 5x64x64 (3 channels for state, 2 channels for the upsampled action) images into a vector with 512 dims following a CNN architecture widely used in RL papers and frameworks [24][25]. Then it will get through a 3-layer MLP with 256 dims latent space to yield the Q value.
• Policy Network: The similar 3-layer CNN based encoder can embed the input of 3x64x64 images into a embedding shaped 32x4x4 (still 512 dims). Then it will get through a 3-layer Deconv to yield grid-like action (2x8x8 or 2x16x16).

In the residual training stage, the policy network’s input channel will also be 5 (3 channels for state, 2 channels for the upsampled coarse action).

[20] Jangir et al. (2022). Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2), 3046-3053.

[21] Zhang et al. (2019, May). Solar: Deep structured representations for model-based reinforcement learning. In International conference on machine learning (pp. 7444-7453). PMLR.

[22] Cho et al. (2021). Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4641-4650).

[23] Dai et al. (2023). Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807.

[24] Mnih et al. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

[25] Raffin et al. (2019). Stable baselines3.

Thank you for your helpful comments. We respond to your comments below.

There is no adequate comparison with classical robot design baselines, e.g. Bayesian optimization, genetic algorithms, etc.

Thanks for your suggestion about the baselines. While co-design is closely related in our related work section, we’d like to clarify that we are working on a novel problem that is drastically different from co-design, rendering the co-design baselines unusable. As we mentioned in the general response, co-design problems fix a robot morphology during execution of the task once the algorithm finds an optimal morphology. In contrast, the goal of our project is to control a type of robot that does not have a fixed morphology. In our problem, the slime robot has to actively control and change its morphology throughout the task. In fact, our robot always begins with a circle or square shape in any task.

Despite this, we added two more related baselines and corresponding experiments following Reviewer UEFW’s suggestion. We take state-of-the-art algorithms in the field of articulated reconfigurable robots, Modular Policy [4], and adopt it for highly-reconfigurable robots by using k-means to assign topological structures to robot muscles. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our rebuttal website. Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Is the action space or environment 2D or 3D?

The action space is represented as a 2-D grid, although our simulator can be easily extended to 3D.

Why use 2D images as raw states for policy instead of using ground truth robot states(i.e., exact morphology parameterizations)?

We choose image observations because pixels are the most general representation of tasks. e.g. Millions of video games are played by humans through pixels; typical real robot manipulation tasks in RL literatures. This has been widely recognized by the deep reinforcement community [5] [6], which considers control from pixels as a more challenging problem [7] [8].

A second reason we consider pixel observation is its potential for real-world transfer. Slime robots, for example the real-world implementation [9] [10], are highly unstructured and hard to obtain an intrinsic state. Pixel observation not only addresses the challenge for us but also organically capturess external world.

---

[4] Pathak et al. (2019). Learning to control self-assembling morphologies: a study of generalization via modularity. Advances in Neural Information Processing Systems, 32.

[5] Mnih et al. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

[6] Shah et al. Rrl: Resnet as representation for reinforcement learning. arXiv preprint arXiv:2107.03380.

[7] Laskin et al. (2020, November). Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning (pp. 5639-5650). PMLR.

[8] Laskin et al. (2020). Reinforcement learning with augmented data. Advances in neural information processing systems, 33, 19884-19895.

[9] Sun et al. (2022). Reconfigurable magnetic slime robot: deformation, adaptability, and multifunction. Advanced Functional Materials, 32(26), 2112508.

[10] Wang et al. (2023). Reconfigurable Liquid‐Bodied Miniature Machines: Magnetic Control and Microrobotic Applications. Advanced Intelligent Systems, 2300108.

Thank you for your feedback! We sincerely wish our written justification provides additional insight to you in case we cannot get additional results done in time.

extend the 2D soft robot simulator to 3D ... more beneficial to claim directly that this simulator can be extended to 3D and how to

Thank you for your question and suggestions! Our physics simulation is coded with Taichi, which simulates 2D and 3D alike with same equations and algorithms (MPM). A widely used piece of sample code for 3D MPM simulation can be found here, (it also demonstrates how to make taichi mpm differentiable) which shares the same material equations as 2D code sample except you change things like 2D rotation matrix to 3D and change 2 loops from 3 loops; even the variable names are shared. One can put these two files side-by-side and translate our simulation in 3D without much problem.

detailed comparison with the current soft robot simulator We provide the suggested table below. Since we are not a co-design benchmark like the one you mentioned and we cited, our environments uniquely allows and requires robots to actively change their shape to accomplish a task (initial shape is always sphere or cube). We think it's an amazing suggestion will include it in our camera ready.

Visual RL is important, but I didn't see the rationale of using pixel observation for robot design

This is because we are not doing robot design, but learning to control a new type of robot! The reason why early RL researchers use state is that control from pixel is hard [7][8], and early algorithms can hardly work unless one gives it the state. From 2018, with the emergence of a good visual-motor-control algorithm, RL community and robot learning researchers have started criticizing state-observation as they can hardly transfer to real world [5] [6] [7] [8].

On the other hand, this was not a problem for co-design community, many of whom also do computer graphics and material science, fields which assume materials can be simulated and analyzed in simulators (and often differentiable). Under such assumption, one use computer algorithms to optimize for a design in a simulator before one can 3D print or manufacture a robot. However, to our best-knowledge, such robots cannot be controlled in real world due to state-spaced policy.

While we recognize the merit of both school of thoughts in the problems they care about, our goal aligns better with RL & robot learning community.

We will try our best to run state-based baselines before rebuttal deadline, though it may take a while to train enough seeds. Still, many important works in RL [5][6][7][8] point out that control from image observation is much harder and useful for real-world.

2D pixel observation can be ambiguous in 3D environment

Good question. To solve this, multi-camera observation has become standard in real-robot robot learning research, especially for robotics manipulation. For example, a popular imitation learning method.

The lack of our differentiability is from our motivation.

Thanks for providing the helpful context and references!

Our lately added GNN Modular Policy baseline on our rebuttal website uses state space observation. It is the state-of-the-art method to leverage structure of state space in a closely related problem. As you can see from figure 1.2, this baseline method is actually much lower than us.

I think the more important comparisons are:

1. your model with accurate state vs. your model with pixel observation;
2. baselines with accurate state vs. baselines with pixel observation.

Because those comparisons are more for understanding the problem setting than for your method justification.

Dear reviewer,

We will try our best to include these results.

We really appreciate the time you spent on understanding our work. If you don't feel comfortable enough raising the scores, please consider raise your confidence for our clarifications!

Besides the evaluation metric on reward, if more metrics can be used to evaluate real-robot applications, it will be appreciated, such as the successful rate or time efficiency.

Thank you for the feedback. First we’d like to clarify that our provided metric, maximum episode reward and sample efficiency curves are usually regarded as the most fundamental metrics in the DRL community. In fact, a lot of highly cited DRL papers choose sample efficiency curve as the only metric [17][18]. This is because metrics like success rate can only be defined in tasks like manipulation, while a task like running forward doesn’t clearly have a threshold called “success”. Instead, designers may only want the robot to run as far as possible instead of letting it stop at a point.

Despite this, we report the success rate of some manipulation or reaching tasks whose success can be clearly defined. In addition, we also provide the wall-clock-time metric for reference. In figure 3 of our rebuttal website), we report these metrics of our algorithm & baselines in two challenging manipulation tasks. We hope such information can give you readers more insight and we are happy to provide additional results if you have additional questions!

As one of the main challenges proposed to address in this paper, the justification of lifetime adaptation is insufficient in the experiments. For example, if the approach can adapt to noise or actuator failures during the lifetime operation?

Thanks for your suggestion. We follow your suggestion and add 2 additional sets of experiments that emulate actuator failure and observation failure.

Regarding noise adaptation, we’d like to clarify that we already added gaussian noise to action output when getting our result in the main paper, to emulate noisy control. The SAC algorithm naturally optimizes expected return under high entropy (noisy) actions, which explains why our result is already good in the paper.

Our two added experiments randomly zeros out 20% of the action output / observation input respectively when evaluating our expert policy. This emulates actuator / sensor failures respectively. Figure 4 on our rebuttal website shows their performance as a ratio to original performance. As shown in the figure, our performance is negligibly impacted (<10%) by action failure except 1 out of the 8 tasks, DIG, which shows a performance drop of only 30%. The policy also seems to show some robustness to sensor failure from the plot, a less common setting.

Even though the simulation experiments are impressive, there is still a gap between applying the proposed work to real-world robots, as shown in the demo video. An explanation of how to extend the work from simulations to real-world robots will strengthen the paper significantly.

Thanks for your comment. In our project, we tried to mimic the settings of real-world slime robots implementation in [19]. In this paper, the researchers use cameras and magnetic fields to demonstrate basic movements of the slime robot. In our benchmark, we adopt the gravity field simulation from Taichi to simulate magnetic fields and adopt the same real-world robot observation hardware researchers use. While we recognize the current framework has its assumptions, we believe our implementation is well-aligned with the hardware progress of real-world magnetic robots, which are bottlenecked by algorithms and benchmarks to which our paper aims to contribute.

---

[11] Mnih et al. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

[12] Raffin et al.(2019). Stable baselines3.

[13] Huang et al. (2021). Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.03311.

[14] Bhatia et al. (2021). Evolution gym: A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34, 2201-2214.

[15] James et al. (2020). Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2), 3019-3026.

[16] Pathak et al. (2019). Learning to control self-assembling morphologies: a study of generalization via modularity. Advances in Neural Information Processing Systems, 32.

[17] Hafner et al. (2019). Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.

[18] Laskin et al. (2020). Reinforcement learning with augmented data. Advances in neural information processing systems, 33, 19884-19895.

[19] Sun et al. (2022). Reconfigurable magnetic slime robot: deformation, adaptability, and multifunction. Advanced Functional Materials, 32(26), 2112508.

Thank you for your helpful comments. We respond to your comments below.

The explanation of the full approach is unclear. For example, multiple variables and terms are not defined or explained in the figures, including vx, vy, and SHAPE in Figure 3.

Sorry for the confusion. Our vx and vy are tracked velocities of particles, rasterized to pixels. SHAPE is the rasterized occupancy of particles. We will update the paper accordingly in the camera-ready version.

What is the dimension for actions?

As we mentioned in section 5.2, the action output is 16x16x2 before interpolation, although the action space itself is infinite dimensional in our formalization.

What are the architectures of encoder, Coarse, Residual policy?

Sorry about the confusion. We follow the natural CNN architecture widely used in RL. The network details are shown below:

• Critic Network: The 3-layer CNN based encoder can embed the input of 5x64x64 (3 channels for state, 2 channels for the upsampled action) images into a vector with 512 dims following a CNN architecture widely used in RL papers and frameworks [11][12]. Then it will get through a 3-layer MLP with 256 dims latent space to yield the Q value.
• Policy Network: The similar 3-layer CNN based encoder can embed the input of 3x64x64 images into a embedding shaped 32x4x4 (still 512 dims). Then it will get through a 3-layer Deconv to yield grid-like action (2x8x8 or 2x16x16).

In the residual training stage, the policy network’s input channel will also be 5 (3 channels for state, 2 channels for the upsampled coarse action).

We have added more detailed illustrations of our method in figure 2 on the rebuttal website.

In the appendix, the paper briefly explains that the core framework is based on existing works SAC and Nature CNN with minor modifications, but it is unclear how to reproduce the full approach.

We will open-source the entire code base upon acceptance. We will also update the paper to include these details.

Given the concern above, the theoretical novelty is not high, as the main theoretical novelty of CFP is the introduction of adding residual action to coarse action.

While our paper is not a ML theory paper, we still believe we have strong technical contributions that can inspire broader research from the community. First, formalize a novel control problem that has never been studied algorithmically into a Markov Decision Problem. This establishes a platform that future researchers can build on. To our knowledge, we are the first paper to let robots control its muscle to change its own morphology, not to mention the slime robot. Secondly, we overcome the challenges of simulation and provide the RL community with a set of fast benchmarks of this interesting but novel task. Our effort on the simulation is non-trivial and such novel benchmarks alone have been traditionally regarded as technical contributions in many top ML conferences [13] [14] [15]. Thirdly, CFP is one of the first RL algorithms that uses a 2D action space. Many design choices here such as fully convolutional network or coarse to fine are largely overlooked by the RL community, which traditionally focuses on unstructured action space while our insight brings attention to such 2D structure. We believe our formalization, benchmarks and algorithm together shall constitute technical contributions above the acceptance threshold.

The paper lacks comparisons with existing state-of-the-art methods.

Thanks for your suggestion. We added two more related baselines and corresponding experiments following Reviewer UEFW’s suggestion, although we’d like to clarify that these baselines have to undergo significant adoption as we have a very novel task.

We take state-of-the-art algorithms in the field of articulated reconfigurable robots, Modular Policy [16], and adopt it for highly-reconfigurable robots by using k-means to assign topological structures to robot muscles. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our rebuttal website. Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Dear Reviewer NgHH,

We first thank you again for your comments and suggestions. 

In our earlier response, we have provided detailed clarification based on your questions about the method details. In addition to that, we also added additional method details in appendix section E and will release the code to make this fully reproducible.

We have also conducted additional experiments with strong baselines to illustrate the effectiveness of our proposed method on our rebuttal website and we also compared the performance loss when faced with observation or action failures following your great suggestion. Furthermore, we have also listened to your useful advice to put up more metrics such as time efficiency and success rate to comprehensively evaluate our method. Finally, we have provided illustrations about how we can extend our project with real-world settings.

As we are ending the stage of the author-reviewer discussion soon, we kindly ask you to review our revised paper and our response and reconsider the scores if our response has addressed your concerns. 

If you have any other questions, we are also pleased to respond. We sincerely look forward to your response.

Best wishes!

The authors

Due to such fundamental differences in the problem, many of the suggested co-design baselines such as Bayesian Optimization and Genetic Algorithm [2] cannot be used on reconfigurable robots, as we are controlling morphological changes by muscles rather than searching for a mophology.

I'm confused about the essential differences between controlling morphological changes and searching for a mophology.

My understanding of controlling morphological changes is continuously searching for optimal morphologies. Can the former one be considered as a continuous loop of the latter one?

Glad to make an important clarification here.

Searching for a mophology refers to the design process of co-design. Once your algorithm finds out a design for a robot, at the frame 1 of your task, your robot is already in that shape. On the other hand, in our problem setting, our robot always have a shape of sphere in frame 1, and the robot need to figure out a sequence of muscle actions to bend it into a target shape.

Imagine you are making sculptures of animals. Search for a mophology is like, you imagine the a shape in your mind and that's it. Controlling the mophology is like, you not only need to imagine the shape, but also need to use your hand to lift tools, infuse water, mix up the clay and take all these physical actions to create that sculpture.

Some co-design papers also talks about control, which is something after you make the sculpture (e.g. you are making sculptures of animals and you want to evaluate whether the animal can run). But their design itself has the difference we mentioned above. The control in co-design paper assumes someone already magically made the sculpture for you and then you do something on top of it, which is very different.

So co-design is getting a shape with brain only, while our setting a robot needs both brain and muscle.

My understanding of controlling morphological changes is continuously searching for optimal morphologies. Can the former one be considered as a continuous loop of the latter one?

Not exactly as I described above. As I mentioned above, co-design's design process never requires your muscle movement.