SQT -- rough conservative actor critic

1. The contributions of this paper are insufficient. The primary contribution is simply the addition of the standard deviation of ensemble networks to the Q-target values. The authors should consider further how the experimental effect differs for different penalty terms, or give an explanation for why the standard deviation is used as a penalty term, and not some other difference, e.g. variance. In addition, the authors need to conduct ablation experiments on the hyperparameter in front of the penalty term to study the effects of different hyperparameters on the test results.
2. The use of ensemble Q estimators is not particularly novel and the authors did not cite the relevant papers in Background section, such as some related ensemble learning methods.
3. The citation format in the paper is inconsistent, and the use of abbreviations and table formatting are irregular, affecting the paper's readability and adherence to standards. Such as Eq. (10) first appears in the pseudocode without any explanation in the text. It is recommended to provide an explanation when the formula is introduced for the first time. Additionally, when abbreviations such as 'RL' and 'MDP' appear for the first time, the full terms also should be provided.
4. The experiments in the paper only compare a few benchmark algorithms and do not include more recent reinforcement learning algorithms. It is recommended to include additional benchmark comparisons and statistical analyses, such as DreamerV2 and REM.

1. I didn't like the way the paper is structured. The method is described in section 8 while most of the results are presented before. I could not understand what is the proposed approach until the page 8. I strongly recommend authors to put the description of their approach (what is SQT operator) in section 4.
2. Unclear motivation and description of the main approach. In equation 20 authors demonstrate the formula for their main approach, if I understand correctly. However what are s' and a' under the mean part? Those are supposed to be the next state and action in the other part of equation. Is it just a random set of states and actions from the replay buffer? If so, I do not see the motivation behind the schema. I fish authors clarify and motivate it better. I.e. why random set of state-actions is useful for learning Q(s, a) for a particular (s, a) pair? In case of SQT_{tuple} I have no questions as we use only the following pair.
3. Questionable claim about the superiority of SQT when compared to SQT_{tuple}. It outperforms only on one task out of four. It looks very suspicious and the claim " This highlights the advantage of SQT’s global penalty mechanism, which leverages the wisdom of the crowd principle by applying a single penalty based on the batch standard deviation across all state-action pairs" seems too strong to me. I consider it is as critical to show on more tasks, otherwise the efficiency is very questionable.
4. Limited evaluation. I'm not strongly aware about environments which are used for online RL approaches now but at least there should be a Hopper MuJoCo environment. Are there any more environments where authors can demonstrate that SQT is better than SQT_{tuple}?
5. No statistical significance is reported. There are only mean scores in the tables and no learning curves.
6. No offline RL results which could be very relevant here.

• This exact idea has already been published before: "Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters" (Ghasemipour et al, NeurIPS 2022). See its Shared Targets (Method 2).
• Experimental results are extremely weak.

• Most important: Lack of in-depth analysis of the effect of the proposed approach on bias reduction. The only thing shown in the paper is the performance, give the proposed change is quite simple, the paper can benefit from more related experiments to help understand the issue better. For example, one way to do this is to show how the amount of bias changes for different algorithms, you can get the Q predictions and estimate the true Q values with MC return (as done in the REDQ paper you cited), or you can look at how Double Q learning paper ("Deep Reinforcement Learning with Double Q-learning") did it.
• Lack of technical details: there is no adequate discussion of hyperparameters and hyperparameter sensitivity, and other details such as how much computation the method requires compared to the baselines?
• There are other algorithms that utilize the uncertainty of Q ensembles to help reduce bias, such as TQC ("Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics") and REDQ ("Randomized Ensembled Double Q-Learning: Learning Fast Without a Model"), how does the proposed approach compare to these methods?
• Since this paper focuses on empirical results, the paper can be made stronger by providing more extensive results, such as comparing to more baselines and test on more benchmarks (e.g. other MuJoCo environments, deep mind control, Atari).

Minor issues:

• Line 309 you may want to introduce what UTD stands for here.