ABNet: Attention BarrierNet for Safe and Scalable Robot Learning

Theoretical Concerns

1. Linear Combination of Policy Outputs

   ABNet generates its final control output by linearly combining the outputs from multiple heads. However, there are concerns about whether such a linear combination is effective for complex nonlinear problems. Many real-world environments involve significant nonlinearity and complex interactions, which may not be fully captured by linear combinations.

2. Generalization of Safety Guarantees

   While the paper provides theoretical guarantees that the combined outputs ensure safety, there is insufficient explanation regarding whether these guarantees hold in novel situations not covered by the training data. If the safety constraints do not generalize across all scenarios, the model may malfunction in unexpected environments.

3. Gap Between Mathematical Models and Reality

   ABNet relies on mathematical optimization (specifically Quadratic Programming, or QP) to implement safety control. However, real-world uncertainties, noise, and sensor errors may reduce the model’s performance. There is a concern about whether the mathematical models, which do not account for such uncertainties, can achieve similar performance in practice.

Experimental Concerns

1. Gap Between Simulation and Real-World Environments

   Most of the experiments in the paper were conducted in simulated environments. However, there are significant differences between simulations and real-world conditions. For example, sensor errors and unexpected scenarios, which may not be fully replicated in simulations, could affect safety in practice.

2. Interpretation of Performance Metrics

   The paper presents metrics on rewards and safety but may have overlooked potential trade-offs between safety and performance in specific tasks. For instance, to achieve high rewards, the model might subtly violate some safety constraints, but this trade-off has not been analyzed in detail.

3. Limitations of Comparative Experiments

   While ABNet's performance was compared with several existing models, there is a possibility that the comparison was not exhaustive. The inclusion of more recent reinforcement learning or control models would enhance the reliability of the results.

4. Overfitting Risks

   Since ABNet trains each head individually and combines them linearly, there is a potential risk of overfitting to the training data. This could limit the model’s generalization to unseen scenarios, leading to performance degradation in untested environments.

Comprehensive Concerns

• Generalization and Data Limitations: If the training data and safety constraints are limited, the model might struggle to handle the wide range of situations encountered in real-world applications.
• Real-Time Control Challenges: The QP-based optimization required for safety control may not be suitable for real-time control due to computational delays. The paper lacks a detailed discussion of latency issues, which are critical for real-time robotic control.
• Probabilistic Safety Guarantees: If safety is guaranteed only probabilistically or as an expected value, rare but critical safety violations could occur. The paper does not adequately address how to handle these rare but potentially catastrophic violations.

These theoretical and experimental concerns are crucial for evaluating the reliability of ABNet’s proposed methodology and results. Future research should focus on conducting experiments in more diverse environments and incorporating complex real-world scenarios. Additionally, further studies are needed to ensure real-time performance and provide clearer safety guarantees.

1. About the definition of attention

1a. How is the self-attention used for ABNet? Usually, self-attention refers to the one in the transformer. I think, in general, ABNet is not restricted to being a transformer. So how was it used for the ABNet? Is it used for both state-based task and vision-based task?

1b. A follow-up question: Is Fusion also part of the attention? If so, I think it is not the usual self-attention. Normally, self-attention calculates the score based on the softmax between K and Q. In this paper, it is simply a weighting coefficient. The fusion looks more similar to model ensemble rather than self-attention.

2. About the effectiveness of attention

2a. I wonder whether the approach's effectiveness comes mainly from enlarging model size instead of using this attention mechanism. Can we have some experiments that enlarge the size of the BNet to the same size as those best models, e.g., ABNet-10 or ABNet-100, and compare this large BNet to the same-size ABNet, w.r.t. the existing benchmarks? It would be appreciated if the model parameter size for each method could be a column in the table.

3. About the experiment setting

3a. Is imitation learning used for the experiment? If so, do we assume a safe enough reference controller should be known?

3b. The vision-based task seems the most interesting one, and I wish to know more about it. If I understand correctly, the x space here is the image space. How are the dynamics defined (or are they even unknown), and are they control-affine?

3c. I notice in the setting, "with uniformly distributed noise, 10% of the input magnitude in testing". Why are they needed for the testing? Is it for the claim of robustness of the control? I wonder what the performance will be if we simply make the training and testing dynamics the same.

4. About theoretical guarantee

4a. A general problem with the learning-based CBF is that they are guaranteed to be a CBF only when they are trained very well to satisfy the forward invariance property. If out-of-distribution states happen, or the neural CBF fails to fit on the training dataset, then it is not a theoretical CBF anymore. Just as a discussion, does this work circumvent this problem? Meanwhile, I don't think this is a big issue since almost all neural CBFs have this problem.

1. Would the method still be effective in environments that require sophisticated contact handling? How might this approach be extended to humanoid robots, especially in scenarios involving complex external interactions, such as full-body dynamics with external force feedback?

2. To my knowledge, large transformer models are increasingly applied in autonomous driving scenarios involving dynamic external agents, as referenced in [3]. This approach uses transformers as a backbone for safety planning based on sampled trajectories, ensuring collision-free paths while closely resembling NMPC planners or human driving behaviors. How do the authors view the online planning, and could barrier functions be integrated to enhance safety?

3. The method’s applicability appears limited by the requirement for a known, differentiable constraint function. In many real-world scenarios, defining such a function can be highly complex or may even resemble a black-box system. For example, ensuring that large language models avoid generating outputs contrary to human intentions necessitates defining a "human-safe" constraint function, which is inherently difficult.

[3] Planning-oriented Autonomous Driving

1. How does ABNet handle situations where the safety constraints of different heads conflict? For example, if one head focuses on obstacle avoidance and another on goal attainment, how are these constraints balanced?

2. Could you provide further clarification on the choice of penalty functions in the HOCBFs? Specifically, how sensitive is the model's performance to different choices of these functions?

3. Has any analysis been done on the computational cost of scaling up ABNet to a large number of heads, especially in real-time applications like autonomous driving?