Robust Automatic Modulation Classification with Fuzzy Regularization

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

Q1: Can you provide a more formal and detailed explanation of the adaptive gradient mechanism used in Fuzzy Regularization? This would help clarify how it dynamically adjusts during training.

RE:Thank you for your question, it helps to articulate our work more clearly. We have explained the establishment of Eq. (6) in more detail and hope that this will help you to understand the automatic gradient mechanism in FR. First choose the first k predicted values of each sample to calculate the degree of fuzziness ($\mathrm{M}$), and the theoretical maximum value $\mathrm{M}\_{\mathrm{max}}=\left(\sum\_{i=1}^{k}\hat{y}\_{i}\right)^{2}-{k\mu}^2$, minimum value $\mathrm{M}\_{\mathrm{min}}=k\times\left(\frac{\sum\_{i=1}^{k}\hat{y}\_{i}}{k}\right)^{2}-{k\mu}^2$, and mean value $\mu=\frac{\sum\_{i=1}^{k}\hat{y}\_{i}}{k}$ for a single sample are known. Then $\mathrm{M}$ can be expressed as: $\mathrm{M}=\sum\_{i=1}^\mathrm{k}\left(\hat{y}\_{i}-\mu\right)^2=\sum\_{i=1}^\mathrm{k}\hat{y}\_{i}^2-{k\mu}^2$; we focus only on the distribution state, so by normalizing the formula we get $\mathrm{M\_{norm}=\frac{M-M\_{min}}{M\_{max-M\_{min}}}=\frac{\sum\_{i=1}^{k}\hat{y\_{i}}^{2}-k\times(\frac{\sum\_{i=1}^{k}\hat{y\_{i}}}{k})^{2}}{(\sum\_{i=1}^{k}\hat{y\_{i}})^{2}-k\times(\frac{\sum\_{i=1}^{k}\hat{y\_{i}}}{k})^{2}}}$.The automatic gradient mechanism is realized by the log(*) function to correct the gradient of $M\_{norm}$. The absolute value of the gradient of the log(*) function decays symmetrically when it deviates from the central axis, which can satisfy the second definition of the design of FR. The final loss of a single sample is: $\mathrm{Loss}=\log(\mathrm{M}\_{\mathrm{norm}})=\log\left(\frac{\sum\_{i=1}^k\hat{y}\_i^2-k\times\left(\frac{\sum\_{i=1}^k\hat{y}\_i}{k}\right)^2}{\left(\sum\_{i=1}^k\hat{y}\_i\right)^2-k\times\left(\frac{\sum\_{i=1}^k\hat{y}\_i}{k}\right)^2}\right)$.

Q2:How does the FR mechanism compare to label smoothing or other entropy-based regularization methods in terms of performance and robustness? Would it be beneficial to combine these techniques?

RE:We would like to thank the reviewers for their valuable comments. First we define Ent to denote the entropy-based method, LS to denote the label smoothing method, and FRLS to denote the joint training of the label smoothing and FR methods. NF_* indicates that the noise factor of the dataset is *.For the sake of intuition and brevity, the data recorded is the performance difference between the two methods, e.g. Ent_FR is the value obtained by subtracting the entropy-based model accuracy from the FR model accuracy in the original article.

Through Table I we find that the performance difference of entropy-based methods compared to WF methods is both good and bad. We analyze that the reason for the deterioration of performance may be due to the gradient update of the entropy function and other problems, which is one of the difficulties solved in this paper. Although the performance of the entropy-based method can be close to the FR method in some tasks, the performance is weaker than the FR method in general tasks. This also shows that the FR method is better than the entropy-based method.

Table 1

With Table 2 we find that the FRLS models generally outperform the models trained with label smoothing loss, which further demonstrates the FR validity. Meanwhile, we find that the model trained by FRLS joint loss does outperform the model trained by FR supervision on some tasks. This may be due to the fact that LS optimizes the target labels, which make the values of non-target classes non-zero.This helps the model to learn more information from other classes. FR mainly focuses on the model's predictive distribution information. The two focus on different information, so the performance of the model can be further optimized when trained jointly. We will continue to study the joint training strategy in depth.

Table 2

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1: The work appears to bear some resemblance to curriculum learning. Could the authors provide a detailed explanation of the similarities and differences between these two methods?

RE: This is very good question. There are two differences between FR and curriculum learning. Firstly, the training strategy is different. Curriculum learningis a two-stage learning strategy: learning the easy samples then the difficult samples [1]. FR did not have this strategy and focused on marginal samples(which can be regarded as the difficult samples in course learning) from the beginning. The second and biggest difference is that the both of them utilize different information. Since FR can measure the degree of predictive ambiguity of the samples, it utilizes more information about the predictive distribution of the samples than course learning.

[1] Liu Y, Wang J, Xiao L, et al. Foregroundness-aware task disentanglement and self-paced curriculum learning for domain adaptive object detection[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023.

W2: The backward derivation of this regular derivative is not seen in the text, but it is important for the subsequent optimization of the model, and it is hoped that the author can add this part of the content.

RE: In order to facilitate the detailed explanation of the subsequent derivation process, we provide a more detailed additional description of Eqs. (7)(8), which is supplemented as follows:
$\mathrm{F}(\hat{y}\_{j})=\frac{\sigma}{T(\hat{y}\_{j},\tau)\sqrt{2\pi}}exp\left[-\frac{\sigma^2}{2}log\left(T\left(\hat{y}\_{j},\tau\right)\right)^2\right]$ $\mathrm{s.t.}\quad\begin{cases}\sigma=-\frac{1}C\sum\_{j=1}^c\frac{\sum\_{i=1}^{k}\widehat{y}\_{ji}log(\widehat{y}\_{ji})}{log(\tau)}, \\\T(\widehat{y}\_{j},\tau)=\frac{\sum\_{i=1}^{k}\widehat{y}\_{ji}^{2}-k\left(\frac{\sum\_{i=1}^{k}\widehat{y}\_{ji}}{k}\right)^2}{\left(\sum\_{i=1}^{k}\widehat{y}\_{ji}\right)^2-k\left(\frac{\sum\_{i=1}^{k}\widehat{y}\_{ji}}{k}\right)^2}.&&&\end{cases}$

where $\hat{y}\_{j}$ denotes the jth sample, $\hat{y}\_{ji}$ denotes the probability that the sample belongs to class i, C denotes the sample size of the batch, $\tau$ denotes the $\tau$ classification task and k denotes the selection of the first k values.

1. Calculating derivatives of intermediate variables

1.1 The derivative of $\sigma$ with respect to $\widehat{y}\_{ji}$

$\frac{\partial\sigma}{\partial\hat{y}\_{ji}}=-\frac{1}{\mathrm{Clog}\tau}\cdot\frac{\partial}{\partial\hat{y}\_{ji}}(\hat{y}\_{ji}\mathrm{log}\hat{y}\_{ji})=-\frac{\log\hat{y}\_{ji}+1}{C\log\tau}$.

1.2 The derivative of $T(\widehat{y}\_j,\tau)$ with respect to $\widehat{y}\_{ji}$

Let $S\_1=\sum\_{i=1}^k\hat{y}\_{ji}, S\_2=\sum\_{i=1}^k\hat{y}\_{ji}^2$.
Then $T=\frac{S\_2-\frac{S\_1^2}{k}}{S\_1^2-\frac{S\_1^2}{k}}=\frac{kS\_2-S\_1^2}{(k-1)S\_1^2}$, we let the numerator in $T$ be N and the denominator D.

So $\frac{\partial T}{\partial\hat{y}\_{ji}}=\frac{\partial N/D}{\partial\hat{y}\_{ji}}=\frac{\partial N}{\partial\hat{y}\_{ji}}\cdot\frac{1}{D}-\frac{N}{D^2}\cdot\frac{\partial D}{\partial\hat{y}\_{ji}}$.

1.2.1 The derivative of the molecule N

Due $\frac{\partial S\_2}{\partial\hat{y}\_{ji}}=2\hat{y}\_{ji}, \frac{\partial S\_1}{\partial\hat{y}\_{ji}}=1$. So $\frac{\partial N}{\partial\hat{y}\_{ji}}=k\cdot\frac{\partial S\_2}{\partial\hat{y}\_{ji}}-2S\_1\cdot\frac{\partial S\_1}{\partial\hat{y}\_{ji}}=2k\hat{y}\_{ji}-2S\_1$.

1.2.2 The derivative of the denominator D

$\frac{\partial D}{\partial\hat{y}\_{ji}}=(k-1)\cdot2S\_1\cdot\frac{\partial S\_1}{\partial\hat{y}\_{ji}}=2(k-1)S\_1$.

1.2.3 The final derivative of $T(\widehat{y}\_{j},\tau)$ with respect to $\widehat{y}\_{ji}$

$\frac{\partial T}{\partial\hat{y}\_{ji}}=\frac{2k\hat{y}\_{ji}-2S\_1}{(k-1)S\_1^2}-\frac{(kS\_2-S\_1^2)\cdot2(k-1)S\_1}{(k-1)^2S\_1^4}=\frac{2k(\hat{y}\_{ji}S\_1-S\_2)}{(k-1)S\_1^3}$.

2. Chain rule for derivation

Due $\frac{\partial\ln F}{\partial\hat{y}\_{ji}}=\frac{1}{\sigma}\frac{\partial\sigma}{\partial\hat{y}\_{ji}}-\frac{1}{T}\frac{\partial T}{\partial\hat{y}\_{ji}}-\sigma(\log T)^2\frac{\partial\sigma}{\partial\hat{y}\_{ji}}-\frac{\sigma^2\log T}{T}\frac{\partial T}{\partial\hat{y}\_{ji}}$.

So $\frac{\partial F}{\partial\hat{y}\_{ji}}=F\cdot\frac{\partial\ln F}{\partial\hat{y}\_{ji}}=F(\hat{y}\_j)\cdot\left[\frac{\partial\sigma}{\partial\hat{y}\_{ji}}\left(\frac{1}{\sigma}-\sigma(\log T)^2\right)+\frac{\partial T}{\partial\hat{y}\_{ji}}\left(-\frac{1}{T}-\frac{\sigma^2\log T}{T}\right)\right]$.

Other Comments Or Suggestions

We will correct the numbering problem in the appendix.

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1: Does FR sharpen misclassified samples?

RE: Thanks for your valuable comment. We can suppress this phenomenon by adjusting the hyperparameters $\gamma$. In the training process, the model is trained with the guidence of both the FR and the cross-entropy function. When the FR originally misclassifies the sample, its corresponding sample' prediction probability value becomes sharp. Although this leads to the decrease of the FR value, but at the same time, it will also lead to the increase of the value of the cross-entropy function. Hence, we can inhibit the occurrence of this phenomenon by selecting an appropriate $\gamma$ value. That is to say, the decrease of the FR loss is smaller than the increase of the cross-entropy loss when the wrong prediction becomes sharp.

For the appropriate value, we have pointed out in Section 4.6 of the paper that the proposed model usually performs better when the FR canonical value is two orders of magnitude different from the value of the cross-entropy function.

W2: Has the author considered that FR regularization might lead to overconfidence issues?

RE: We thank the reviewer for the insightful comments. The work [1,2] shows that overconfidence mainly targets the following two types of samples. The one is that the model's predicted probability value for the class is very high, but the classification result is wrong. And the other is that the model's predicted probability value for the class is very high, but the actual accuracy is much smaller than the predicted probability value.

Firstly, for the first class of samples, we have explained in W.1 that the generation of this class is suppressed by choosing an appropriate hyperparameter $\gamma$. For the second type of samples, we have introduced an adaptive gradient mechanism in the design of FR. This mechanism ensures that the gradient returned by the FR decreases as the value of the predicted probability increases. As a result, the model's predicted probability value for the sample does not always keep increasing. In summary, FR does not lead to overconfidence issues.

[1] A. Roitberg, et al. Is My Driver Observation Model Overconfident? Input-Guided Calibration Networks for Reliable and Interpretable Confidence Estimates, in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 25271-25286
[2] D Wu, et al. The overconfident and ambiguity-averse newsvendor problem in perishable product decision[J]. Computers & Industrial Engineering, 2020, 148: 106689.

W3: The parameter k in the regularization proposed by the authors is quite significant, but how should I go about selecting this parameter k?

RE: For the choice of the value of $k$, it is suggested to set the maximal number of confusion categories. This is because if a deterministic class has $m$ similar classes, then the model's output value for these $m+1$ classes is closest. This is helpful for FR to accurately measure the degree of predictive ambiguity of a sample.

In this paper, $k$ is set the number of coding categories of the modulation with the most coding types in the dataset. For example, in a quadrature dataset, the categories are: 'QAM16', 'QAM64', 'QPSK' and 'WBFM'. Of these, QAM has two coding types, while the other modulations each have only one coding type. Therefore, the value of $k$ is initially chosen to be 2.

Other Comments Or Suggestions

C represents the number of samples included in a single round of training for the model.
We cite the source of the dataset in Text 4.1Experiments Settings and indicate the download link for the dataset in Appendix A.3.

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1: Section 4.5 lacks experimental validation regarding the time taken for a single training round. Fast convergence in terms of training epochs does not necessarily equate to rapid convergence in actual time. It is hoped that the authors can provide more detailed experiments in this aspect.

RE: Thanks for your professional suggestion. The reviewer's concern: although FR converges faster in terms of training epochs; However, if FR brings too much time consumption in a single round, then FR cannot be considered as effective in speeding up the experimental convergence. According to this suggestion, we conduct addtional experiment and record the shortest training time, the longest training time, and the average training time per round for all rounds in the training process. During the experiments we ensured the consistency of the controllable parameters. The experimental results are as follows:

From the experimental result, it can be seen that there is no significant change in the shortest training time, longest training time, and average training time per round whether the FR is added or not. Combined with the fact that FR can speed up the convergence rounds during training as mentioned in the manuscript, it can be shown that FR can speed up the convergence in actual time.

W2: As dataset sizes continue to expand, could this regularization potentially reduce the training efficiency of the model?

RE: Thanks for the valuable question. The calculation of FR focuses on $\sigma$ and $T(\hat{y}_{i},\tau)$ and their values are obtained by matrix operations. Therefore, this process does not consume too much time. The experiment further proved that FR does not lead to a slowdown in training efficiency. We conducted experimental analyses on three different sized datasets, and recorded the shortest training time, the longest training time, and the average training time per round to the three datasets of different sizes. The specific experimental results are as follows:

The above experimental results show that the average rounds of training time increases by 0.1s and 0.4s on the smaller datasets Data2016a and 2016b, respectively. It is noted that the average time increase is not more than 1s on Data2018a with 20GB, and the increase of the training time for the 200 rounds is not more than three minutes. While the performance is improved by three percentage points. This is acceptable in the field of signal recognition.