Meta-Black-Box-Optimization through Offline Q-function Learning with Mamba Architecture

1. What motivated the choice of Mamba as the sequence modeling architecture? In Table 1, the inference time of Decision Transformer (DT) is comparable to that of Q-Mamba. What primarily drives the performance differences between DT and Q-Mamba—is it due to the additional Q-module? Additionally, how would Q-Mamba compare to QT [1] or QDT [2], which also integrate Q-modules into DT to enhance performance?
2. How does Q-Mamba perform in comparison with other methods in the D4RL environment, a widely used testbed for offline RL methods?
3. Could the authors clarify the problem definition and the necessity of using an offline RL approach in a more accessible way? As I understand, the primary focus of this research is to identify components of the BBO optimization algorithm that enhance generalization.
4. The total training objective is directly defined by the Bellman backup update in Equation 8, differing from the supervised learning objective typically used in sequence modeling for offline RL. How, then, is DT applied in this context? What is the training objective for DT in this setting?

[1] Shengchao Hu, et al. Q-value Regularized Transformer for Offline Reinforcement Learning. 2024 ICML.

[2] Taku Yamagata, et al. Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL. 2023 ICML.

• In the Q-value head, the MLP takes $[s^t, token(a^t_{i-1})]$ along with decision information $\mathbb{O}^t_i$. Can you provide any intuition or empirical evidence for this design choice?
• Can you provide more details about the mamba_block, such as the number of layers used? If only one block of Mamba is used as I understood from the paper, is it fair to compare it with Q-Transformer in terms of model capability? I wonder if we reduce the layer in Q-transformer the performance drops so significantly.
• Why does the online Q-Mamba approach take significantly longer to train compared to the Transformer-based online approach? Is this due to the introduction of the decomposed Q function? While the massive training time for Q-Transformer is understandable, what is the primary reason for the difference in training time here?

1. The paper mentions a performance degradation of Q-Mamba in the online learning setting. Does this mean that offline data is always more effective than online data in MetaBBO?

2. For those optimizers with continuous action spaces, how does Q-Mamba deal with the information loss caused by action discretization?

[Questions and Comments]

• Typo: in Line 143 on page 3, shouldn't "improving" be used instead of "reducing" in the given context?

• Notation for Q-functions: why do the authors denote Q-functions as if they are probability distributions? For instance, in Equation 1, the notation used is $Q(a_{1:K}^t|s^t)$. Wouldn’t it be clearer to write this as $Q(s^t, a_{1:K}^t)$? If there is a specific reason for adopting the conditional notation, an explanation would be helpful.

• Reward signal based on relative improvement: as noted in the weaknesses, why did the authors choose to use relative improvement in objective values, rather than, e.g., an absolute value, as the reward signal? This approach assumes access to an optimization oracle. Have the authors tried alternative reward types? Do they anticipate that the performance of Q-Mamba might degrade if optimal objective values are not incorporated?

• Impact of action dimension ordering: also noted in the weaknesses, the paper would benefit from a discussion on the impact of action dimension ordering on performance. How sensitive is Q-Mamba to this ordering, and do the authors have any insights on how different orderings might affect outcomes?

• CQL regularization: in CQL, the values of OOD actions are minimized rather than regularized to zero. Is there a reason why the authors adopt a zero-regularization scheme in Equation 8? Additionally, if my understanding is correct, the relevant discussion in lines 358-362 should be updated to show how this regularization term differs from what's in CQL.

• Exploration capability: why might Q-Mamba exhibit weaker exploration capabilities than other online baselines? Given that Q-Mamba underperforms its offline variant, it would be interesting to understand what limitations might be affecting its exploration during online training.

• In Table 1, the confidence intervals of RLPSO and Q-Mamba largely overlap. However, the authors state that "Q-Mamba significantly outperforms the online baselines RLPSO, LDP, and GLEET, which control...". This statement could be misleading given the overlapping confidence intervals and would benefit from being rephrased.

• Details of experiments in Section 5.3: can the authors provide a detailed description of the experiments conducted in Section 5.3 and consider adding this information to the appendix?