CoSW: Conditional Sample Weighting for Smoke Segmentation with Label Noise

We thank the reviewer for the constructive comments which helped us improve the quality of our work. In the following, we have provided a point-by-point response to the comments. We adopt different letters to represent the different parts of the question raised. The "W" represents "weakness" and the "Q" represents "question".

W1(1). Not all of the notations are clearly defined or some notations are defined very late (e.g. has been mentioned since Sec. 3.3 but its definition is in Sec 3.4). Not all vectors or matrices have their size defined clearly.

We thank the reviewer for the suggestions, and we will include a list of symbol definitions in the paper.

W1(2). About the notation and reference in Tab. 3a.

We thank the reviewer for the suggestion. "Proto" means to change the regular prediction head into a prototype-based one. “Sample Weight” refers to Eq. 7 and the “Proto Update” refers to Eq. 8. We will also include the reference in Tab. 3a.

Q1. Will the annotation of the re-annotated validation set (the clean validation set) of SmokeSeg and SMOKE5K be released?

Yes, we will release the re-annotated validation set of SmokeSeg and SMOKE5K soon.

Q2. is $\Omega$ in line 37 should be $\omega_{\Omega}$? Where is $\pmb{𝑝}(\pmb{𝑥} _ {𝑛}^{k})$ in Eq.4?

Yes, the $\Omega$ in line137 should be $\omega_{\Omega}$. The $\pmb{𝑝}(\pmb{𝑥} _ {𝑛}^{k})$ in line 155 and 156 should be $\pmb{p}^k$ in the Eq. 4.

Q3. Any insight why different entropies lead to different results?

The reason why different entropies lead to different results can be explained from two perspectives:

1. From the characteristics of the three entropies:

Kapur's entropy and Burg's entropy are both based on Shannon's entropy.

Shannon Entropy ($T(P)$)

• Formula: $T(P) = -\sum_{i=1}^{N} p_i \ln p_i$
• Characteristics:
  • Shannon entropy is a fundamental concept in information theory, used to measure uncertainty or information content.
  • The more uniform the probability distribution (i.e., the closer each ($p_i$) is to each other), the higher the Shannon entropy, indicating greater uncertainty.
  • When an event's probability is 1 (a certain event), Shannon entropy is 0, indicating no uncertainty.

Burg Entropy ($T^B(P)$)

• Formula: $T^B(P) = \sum_{i=1}^{N} \ln p_i$
• Characteristics:
  • Burg entropy differs from Shannon entropy in that it directly sums the logarithms of the probabilities.
  • Since the logarithm function is 0 when the probability is 1 and approaches negative infinity as the probability approaches 0, Burg entropy can heavily penalize extremely low-probability events.

Kapur Entropy ($T^K(P)$)

• Formula: $T^K(P) = -\sum_{i=1}^{N} p_i \ln p_i - \sum_{i=1}^{N} (1-p_i) \ln (1-p_i)$
• Characteristics:
  • Kapur entropy is an extension of Shannon entropy, considering not only $p_i$ but also $(1-p_i)$.
  • This entropy measure is more comprehensive, taking into account the uncertainty associated with both the occurrence and non-occurrence of each event.
  • In extreme cases (i.e., when $p_i$ is 0 or 1), Kapur entropy still provides a reasonable measure, addressing the limitations of Shannon entropy in these scenarios.

By comparing these different entropies, we can see that while they all measure the uncertainty of a probability distribution, they have distinct applications and characteristics. Shannon entropy is the most basic and classic measure, while Burg entropy and Kapur entropy offer extensions and complements for different situations and needs.

2. From the weighting derived:

Shannon's $v_n^k=N^k \frac{\exp(-\gamma || \pmb{x}_n^k-\pmb{p}^k ||_2)}{\sum _ {n=1}^{N^k} \exp(-\gamma || \pmb{x}_n^k-\pmb{p}^k ||_2)}$

Burg's $v_n^k=N^k \frac{(-\gamma || \pmb{x}_n^k-\pmb{p}^k ||_2)}{\sum _ {n=1}^{N^k}(-\gamma || \pmb{x}_n^k-\pmb{p}^k ||_2)}$

Kapur's $v_n^k=N^k \frac{1}{1 + \exp( -|| \pmb{x}_n^k-\pmb{p}^k ||_2 - \lambda_k)}$

(where $\lambda _ k$ in Kapur's are the solutions of $\sum_{n=1}^{N^k} v_n^k=N^k$)

Compared to Shannon's entropy derived weighting (Eq. 7 in paper), Burg's (Eq. 32 in Appendix) lacks the exponential term. However, the exponential term can increase the sensitivity of the model to noisy labels. As a result, in Table 4, the performance of Burg's entropy is slightly inferior to the others.

The relatively poor performance of Kapur's entropy derived weighting (Eq. 37 in Appendix) can be attributed to the fact that we do not have $T_t^K = T_w^K + T_b^K$ for Kapur's entropy, which is valid on Shannon's entropy and Burg's entropy. Hence, for a given classification problem, maximizing the within-prototype Kapur's entropy is different from maximizing Kapur's entropy on the entire dataset. This means we cannot simply consider maximizing the entropy of each prototype independently. But for consistency and comparison, we also use the within-class Kapur's entropy instead of the within-class entropy to design the objective function. This may also be the reason why the performance is not as good as Shannon's entropy. The derivation of the different entropy measures can be found in the Appendix.

Q4. What are the $\lambda$, $\mu$, $\gamma$ used?

In our experiment, we set $\lambda = 0.6$, $\mu = 0.999$, and $\gamma = 0.8$. The $\gamma$ represents the strength of the regularization term in RWE, we provide the performance of models with different values. The details can be seen in the global response PDF file Tab. 1.

Thank you very much for the feedback.

We thank the reviewer for the constructive comments. Below, we have provided a point-by-point response. The "W" for weakness, "Q" for question, and "L" for limitation.

W1. Relationship between the two smoke challenges and CoSW.

The two analyses are to explain why smoke tends to produce noisy labels. We then examine the characteristics of the noisy labels in smoke, finding that it has feature inconsistency due to variable transparency. To address this problem, we propose a conditional sample weighting (CoSW). CoSW aims to employ different weighting criteria for the samples within different feature clusters by constructing the regularized within-prototype entropy (RWE).

W2. Further explanation of $\mu$ and $\gamma$.

The $\mu=0.999$ is for the momentum coefficient. Here we refer to the setting in MoCo (CVPR 2020). The meaning is how much of the previous content is retained with each update. Where $\gamma$ represents the degree of the regularization term in the RWE, the larger the $\gamma$, the more sensitive the RWE is to the distance. We provide experiments with different values of $\gamma$ in the global response PDF file Tab. 1.

W3. About the Eq. 5.

Eq. 5 means combining WE (Eq. 4) and the constraint equation Eq. 2. The original objective without the regularization is to build a uniform assignment, but after incorporating the regularization term Eq. 4, the RWE can determine the noise level of the features.

W4. The explanation of $\varepsilon$.

The introduction of $\varepsilon$ is to prevent the occurrence of singular matrices when inverting the matrix. This is a commonly used technique in LDA. The typical value used is 10e-5. We test 10e-4 or 10e-6 and they also worked for training, but setting it to 0 results in non-invertible matrices during the training.

W5. About the layout and aesthetics of the figures.

We will revise the layout and aesthetics of the figures and tables.

Q1. Can you add details of SmokeSeg and SMOKE5K?

SMOKE5K is a mixed dataset (real + synthetic) and SmokeSeg is an entirely real dataset. The majority of images in SmokeSeg are early smoke. The details of the two datasets are shown below.

*The "Noise Rates" here is the ratio of noisy pixel labels to clean pixel labels (estimated by the validation set).

Q2. Can you add discussions for Tab. 1 and Tab. 2?

In Tab. 1: Our method has a clear advantage in small smoke. As small smoke mostly represents early smoke, being able to recognize early smoke accurately has significance for carrying out rescue work quickly in the real world. In addition, under the same method, we find that CoSW is more suitable for transformer-based backbones. Comparing Trans-BVM and CoSW, the gap is around 1% on ResNet-50, but the gap widens to over 4% on the MiT-B3. A similar phenomenon can be observed on SMOKE5K in Tab. 2a.

In Tab. 2b: The tests are divided into two: 1) Noise ratio (the proportion of data added noise); and 2) Noise degree (the strength of the noise) (detailed in Appendix C). The impact of noise degree is sometimes greater than the noise ratio (e.g., 40% high vs 60% low). Since Trans-BVM is specifically designed for smoke, it achieves better results in low-noise scenarios. As the noise continues to increase, the performance of other methods declines rapidly, but CoSW still maintains the performance.

Q3. How to understand "CoSW does not require clean labels during the training."? Can this method be applied to complete noise? What mechanism?

Many previous methods in noisy labels require a clean validation during the training to guide the model identifying noise. 1) the cost of obtaining clean labels is high. 2) Incorporating validation set into the training makes the pipeline complex. The intuition is that under CoSW, the model can find the common characteristics of smoke (i.e. multiple prototypes), and then determine the noise level of each feature based on them. This process is specifically carried out through RWE. The CoSW aims to employ different weighting criteria in different feature clusters to address the problem of feature inconsistency. Fig. 5b shows the CoSW formation. The CoSW requires the assumption that the majority of pixels have clean labels. When the label mask is completely noisy, the model is also unable to distinguish the noisy labels, because it can not learn which features are the common of smoke. To test the anti-noise ability of CoSW, we design an extreme experiment by directly adding different levels of Gaussian noise to the original labels, until it approaches the complete noise. The results and examples of noisy images can be seen in the global response PDF file Tab. 2 and Fig. 2.

L1. CoSW is not very effective in high smoke pixel ratio.

Since the noisy labels mainly occur at the edges of the smoke, in small smoke, the impact of the noisy labels is greater, while in large smoke (high smoke pixel ratio), the impact is smaller. Therefore, the small smoke is more important and meaningful in the real world, as it can assist in timely rescue. Our CoSW achieves the best performance in small smoke, and Tab. 2b demonstrates that our method has robustness against noise. In the high smoke pixel ratio, the impact of noisy labels is not so great, so the performance of CoSW is not very outstanding, but it is second, with the $F_1$ score close to the first (< 0.2).

L2. CoSW has limitations at low noise scenarios.

The reason why CoSW is slightly inferior to Trans-BVM under a low noise ratio is that CoSW uses the basic segmentation model, while Trans-BVM uses a model specially designed for smoke, with a Bayesian generative model and a Transmission module. When the noise rate increases, the performance of CoSW begins to emerge.

Thanks for your response. Please feel free to let us know if you have any further questions. We are dedicated to further clarifying and addressing any remaining issues to the best of our ability.

Dear Reviewer,

Thank you for carefully reviewing our rebuttal. If you have any further questions, please let us know promptly so that we can resolve them in the remaining time. We hope you will reconsider our score. Thank you again.

Best regards,

The Authors

We thank the reviewer for our paper's positive feedback and constructive suggestions. Here are our responses to the reviewer's comments. We adopt different letters to represent the different parts of the question raised. Where "W" represents weakness and "Q" represents question.

W1(1). L104: What diagrams are the authors referring to?

The “diagrams” here refers to the “research fields”.

W1(2). L127: This is somewhat unclear, what do the authors mean by varieties?

The “varieties” here mean “N random variables”.

W1(3). L137: What do classes mean in the context of this problem could be more clear with a simple example in parenthesis.

Our intuition is to provide a more general expression, where CoSW can also be applied to multi-class tasks.

W1(4). L138 and L196: What is D? The pixel dimension, 3 for RGB?

“D” represents the feature dimension after the pixels have gone through the encoder-decoder. At the same time, it is also the dimension of the prototype. In our experiments, the value of D is 256.

W1(5). Strange grammar/wording: L104-105, L128, L282.

Thank you for the reviewers' suggestions. We explain the issues below:

• L104-105: The research fields include supervised learning, unsupervised learning, and self-supervised learning.

• L128: Shannon entropy can be used to measure the uncertainty of the distribution $T(\Pi) = - \sum _ {i=1} ^ {N} \pi_{i} \ln \pi_{i}$ (with $\sum_{i=1}^{N}\pi_i=1$ constrain).

• L282: To investigate the reasons behind the effects of CoSW, we visualize the formation process of CoSW.

Q1. What classes are the authors referring to in L137? Smoke vs background?

Yes, the classes are smoke and background.

Q2. Could you give some details on the "Real-time" vs "Normal" column of Table 1? Also, what is the runtime or FPS of this approach compared to others, for real-time applications?

The "real-time" refers to using lightweight backbones, such as MiT-B0, AFFormer-B, SeaFormer-B, etc. "Normal" uses the regular backbones, such as ResNet-50, MiT-B3, HRNet-48, etc. Here, we also provide a FPS comparison for the different methods.

Real-time:

Normal:

*Input shape: 512x512; NVIDIA RTX 3090Ti GPU

Q3. This is not a limitation or a flaw, but more a question born from curiosity. Has this approach been applied to any other noisy segmentation problem other than smoke segmentation? The formulation of the approach appears fairly general, meaning that it could potentially have broader use.

We find that the variable transparency also exists in skin lesions, so we supplement the experiments on skin lesions images. The dataset used is the ISIC-2017 dataset. The ISIC 2017 dataset is a well-known public benchmark of dermoscopy images for skin cancer detection. It contains 2000 training and 600 test images with corresponding lesion boundary masks. For the noise setting, we refer to the noise generation approach from the NS-1K dataset. Below we list the experimental results. The sample weighting visualization can be seen in the global response PDF file Fig. 1.

*The specific meaning of the noise setting can be seen in the paper Sec. 5.1 NS-1K part.

Thank you very much for the response.

We understand the reviewer's confusion, but this paper is different from DSA. We would like to clarify the differences between our proposed method and the DSA point-by-point below.

1. The key difference is that CoSW is to construct conditional sample weighting to address the issue of noisy labels, while DSA is to formulate scatter matrices to handle the problem of hard samples. We introduce the information theoretic learning (ITL) to achieve CoSW, which aims to determine sample weighting through uncertainty estimation. Furthermore, we experiment with different entropies, including Shannon's, Kapur's, and Burg's entropy, and provide complete weighting derivations based on each entropy in the Appendix. CoSW only adopts scatter analysis in the loss part, and different from DSA, we further incorporate the sample weighting into the scatter matrix, to prevent metric learning from overfitting under noisy labels.

2. In CoSW, we propose a concept of regularized within-prototype entropy (RWE). RWE can establish independent evaluation criteria for each feature group formed by multiple prototypes. This allows for a more fine-grained determination of the weighting, which is suitable for inconsistent smoke features.

3. For the prototype update, CoSW is also different from DSA. DSA directly uses gradient descent, but gradient descent can easily be affected by noisy labels. We adopt a regularized nonparametric prototype update (Fig. 3 in the paper), in which the new prototype is weighted by all the features that matched the prototype in the previous iteration.

4. The prediction head in DSA and CoSW is different as well. The essence of DSA is to integrate the scatter loss into the original model, still using the classic binary prediction head (2 neurons). But CoSW changes the prediction head to a prototype-based way, where the final output neurons are 2K, with K being the number of prototypes per class.

5. In the experiment, we not only test on real datasets but also create a synthetic noise dataset NS-1K, which has two noisy hierarchies: the ratio of noisy labels to the total labels and the degree of noise. Under this setting, NS-1K can reflect the performance of various models under different noise levels.

Thanks for the reviewer's response. We are delighted to answer any other concerns you may have.

Dear Reviewer,

We sincerely thank you for taking the time to review our paper and for providing valuable comments. Despite some superficial similarities, our method differs significantly from DSA in fundamental ways. We have provided a detailed explanation in our rebuttal.

We cordially invite you to review our detailed rebuttal. If you have any questions, please feel free to contact us. We will do our best to clarify and eliminate the remaining concerns.

Best regards,

The Authors