Your Consistency Model is Secretly a More Powerful Supervised Learning Paradigm for Learning Tasks with Complex Labels

1. The term “SCL” has already been used for supervised contrastive learning, so using a different abbreviation is recommended.

2. While this method shows good performance on the segmentation task, it would be beneficial to explore its applicability on large-scale classification datasets with many classes, such as iNaturalist and Open Images.

3. How much longer does training take compared to standard supervised learning, in terms of both iterations and wall clock time?

4. A key limitation of this method is the increased inference time, particularly in segmentation tasks. Details on the choice of $𝑁_\tau$ during inference, time taken per inference, and wall-clock time compared to a single standard inference on a supervised model would be helpful.

5. Most results in the experimental section were shown using single-step predictions, with limited results provided for multistep predictions. Specifically, Figure 3 shows qualitative results where multistep predictions perform better than single-step, but in Table 1, only results for single-step predictions are provided, and multistep results are absent. Presenting multistep results selectively raises doubts about the performance of multistep predictions.

6. The baseline models used in the experiments are quite outdated (segmentation models from 2017-2019, N-body simulation models from 2015-2017), so it is necessary to verify whether performance improvements still occur when applied to the latest methods.

7. There is no ablation study on the proposed method in the paper. Specifically, Equation 5 uses triadic distance as a loss, but an ablation study is needed to verify whether triadic distance is an essential component. Additionally, an ablation study on the step size in the training process seems also necessary, as it needs to be demonstrated that a large step size is essential, given that only single-step or few-step predictions are actually used during inference.

8. As it takes multiple forward passes for an inference, how does performance compare to ensemble methods, that require a similar number of inferences such as Bayesian model averaging, stochastic weight averaging, or SWAG?

9. Even though the method was proposed for supervised learning with a large number of classes, I wonder what would be the performances for tasks with a relatively small number of classes, e.g. classical classification problems such as imagenet or cifar-100.

10. Definition 2.1 lacks the formal structure expected of a definition, so rephrasing it to clarify the characteristics of "complex labels" would enhance the readability.

11. In the introductory part and motivation, the authors mention information bottleneck (IB), but the derivation of the algorithm is fairly relevant to IB. More discussion or theoretical relationship between the proposed algorithm and IB would be beneficial.

If the rebuttal is satisfactory, I am willing to raise the score.

1. Inconsistencies with Consistency Model Definition: The paper Claims that SCL framework is a nevel training paradigm which embodies concistency model framework, however, I don't think whether SCL framework is Consistency model. Specifically, the model does not receive noisy inputs, which contradicts the typical definition of consistency models, where the output should adapt to noisy changes in the input. Rather, the model produces different outputs based solely on varying time steps t, similar to using t as a prior in a Bayesian model. The framework is much close to Baysian model. The SCL framework gets image and time t, which is similar to get image and prior importance. If the prior is important, then the model should sample the output in much uncertain distribution. In this viewpoint, the sampling algorithm is just baysian inferecne and it can explain, with marginal performance improvements.

2. Inefficiency in Multistep Prediction: The proposed multistep prediction approach raises questions about its efficiency. In the described algorithm, it appears that inference is performed for all time steps t and then averaged, as illustrated in Figure 3, where time step values range from 1000 to 0. This implies nearly 1000 inferences, which seems naively similar to multiple rounds of training and ensembling. Moreover, the performance gains compared to ensemble approaches appear marginal in most cases.

3. Low Reported Performance: Upon examining Table 1, the reported results seem unusually low compared to well-known baselines reported in other prominent papers. For example, MobileNetV2 + SCL shows a significant improvement over the baseline MobileNetV2, but the reported baseline mIoU value of 17.84 seems unusually low, raising concerns about its validity, especially considering that even the 2015 FCN architecture reported an mIoU of 30. This suggests that the baseline performance reported for MobileNetV2dilated is abnormally low. Similar discrepancies are observed with other baselines compared to values reported on platforms like Papers with Code.

• The loss does not seem well grounded

  • The $\lambda_2$-term $\lambda_2 d(f(x,y_t,t),f(x,y_{t'},t'))$ seems redundant with the $\lambda_1$-term $\lambda_1 (d(f(x,y_t,t),y) + d(f(x,y_{t'},t'),y))$ since the $\lambda_1$-term is already mapping $f(x,.,.)$ to $y$.
  • This loss also is disconnected from the standard consistency models definition since it has not self-teaching aspect (not recurrence), which is at odds with the title "Your Consistency Model is Secretly...". What is proposed here does not look like what is classically defined as a consistency model to me. To be more explicit the $\lambda_2$-term is not recurrent since it refers to two independently sampled noise (also called independent trajectories in the paper), and unlike in consistency models there's no 'stop_gradient' indicating that one side is acting as a teacher for the other.
  • I suspect the whole loss could be simplified to $d(f(x,y_t,t),y)$, which basically is, in expectation, the case when $\lambda_2=0$.

• Lack of ablation studies

  • Authors claim that more denoising steps lead to better results but I didn't see any ablation study confirming that claim.
  • The $\lambda_1$ and $\lambda_2$ terms seem important in the loss and yet no in-depth numerical analysis of various combinations is shown, this would help me gain more insight into the necessity of the two terms for the proposed loss.

• Improvements are marginal overall: typical claimed gains are of 1 point (on various metrics and various domains) with one exception for MobileNetV2 which was originally performing a lot worse than other networks in the first place.

• The experiments cover 4 different domains leaving me with a feeling of breadth at the expense of depth.

• The writing is hard to follow in general and there's a fair amount of repetition. In particular the method could be better presented although the inclusion of the algorithm helped a lot to comprehend what is actually being proposed.

1. One shortcoming is that the paper assumed the data provides sufficient information about the labels, but this often happens when there is an abundance of data. It is an assumption that is implicitly made by the method, but was not mentioned at all.
2. The writing of the paper begins and motivates from the perspective of Information Bottleneck. However, from the writing, it’s not exactly clear how the high level intuition connects with the assumptions about the model. Intuitively, I believe the authors are trying to argue that consistency models also make the assumption that the distribution it’s trying to model (i.e. the target distribution here) is also trying to learn a low-dimensional representation, for instance. However, this is not explicitly said or mentioned. In other words, the discussion on IB seems to distract from the discussion rather than helping the discussion. The introduction would have read fine without any mentioning of IB. Some additional comments regarding the connection in the related work section would make the motivation of the work more clear and prominent.
3. While one could argue that consistency models are a recent rising trend, the method is somewhat simply applying consistency models in place of other generative models for learning target distributions. In this sense, there is limited novelty. It can help if the authors can comment more on how consistency model achieves something that previous generative models cannot. This weakness is also listed as a question for the authors below also.