Questions

[Q1. COMPUTATIONAL TIME COMPARISON.] There is real-time performance comparison in Sec. E.2 of Appendix, which shows the comparison between TARSS-Net and some other models, involvingmodel size, calculation amount, real-time performance and other information of in details. In addition, we also add the VIT model inTable S2 as a baseline for comparison, which can also be found inTable 1 of attached PDF for this rebuttal.

[Q2. MINIMAL DIFFERENCE IN PERFORMANCE COMPARED WITH PKCIN-NET.] PKCIn-net makes the classical radar target detection CFAR principle deeply learnable, and proposes a radar-specific target detection operator PeakConv (PKC). The focus of PKC is not on incorporating temporal information, but we include it in the comparison as it is a landmark work on introducing deep learning to RSS. Under the premise of the same model size, TARSS-Net outperforms PKCIn-Net by an average of 2% (on all data-view and three dataset). First of all, given the particularity and difficulty of RSS tasks, a 2% performance improvement is significant, we also illustrate in our responses to R. MgaH how the RSS performance has improved little by little (if you want to learn more, please see response of W2 to R. MgaH). Secondly, PKCIn-Net focuses on making the classical radar detection theory learnable, and TARSS-Net focuses on introducing an efficient temporal relationship modeling method suitable for radar signals. In the later work, we will also consider adapting PKC to TARSS-Net to further improve the performance of the model, since TRAM and PKC are not in conflict due to their different focuses. Both PKCIN-Net and TARSS-Net point the way for the development of the RSS field. That is, compared with directly applying the mature learning method in other field (such as CV, NLP, etc.), designing the learning paradigm suitable for the characteristics of radar signals is more beneficial.

[Q3. WOULD IT BE POSSIBLE TO FIRST FUSE SPATIAL AND DEPTH INFORMATION, AND THEN SUBSEQUENTLY FUSE THIS COMBINED DATA ALONG THE TEMPORAL AXIS?] This is a particularly valuable question. When we originally designed the TRAP module to implement temporal relation measurement and weighted-fusion, we intended to incorporate spatio-temporal information at the same time. However, for the space and depth with high dimensions, it is the most economical choice to do dimensionality reduction and fusion processing separately, i.e., Spatio/Depth-TRAP. If we want to fuse the two dimensions at the same time, it is bound to introduce modules with larger parameters, so we did not explore it in this work. Therefore, in the subsequent network optimization process, considering the complexity of RSS network to facilitate its application, we form two versions of Spatio-TRAP and Depth-TRAP on the premise of ensuring their performance advantages. Your idea is very meaningful, and as mentioned in the limitations Section in Appendix F (L1017), deeply exploration on the learning principle and applicable scenarios of TARSS-Net_D and TARSS-Net_S is needed in future work. Based on this, we look forward to working out an exciting solution to your pertinent question.

We have provided detailed responses to all of your questions. Hope to get your approval on the reply and update the rating. We also look forward to more in-depth communication and discussion with you. Thank you again!

Weaknesses

[W1. LACK OF CLEAR HYPOTHESES AND REASONING.] Sorry for the confusing, and you are right that clear hypotheses and reasoning are important. However, at present, the data-driven learning training of AI models allows researchers to conceptuate and design algorithms at a higher level (the functional and motivated level), which means the classification interface (Modeling functions of network) is no longer expressed in an explicit way by formulas, but is learned implicitly in the network parameters. It solves the problem that the classification interface cannot be shown by existing expressions. In this way, the algorithm performance has been further improved and generalized. The writing of TARSS-Net exactly follows this theoretical system of priori interpretability. Of course, TARSS-Net essentially involves a series of theories and derivations of feature engineering, metric learning, differentiable layer design and so on with radar signal as input. Due to the space limitation and ease of understanding for readers with general AI research background, this paper starts from the high level design and implementation, but the necessary derivations in key parts are preserved.

In addition, this paper gives a detailed introduction on why TARSS-Net is effective for RSS. We have conducted a comprehensive discussion for existing time series modeling paradigms, and analyze their own drawbacks and the factors that make them unsuitable for RSS (see Sec. 2). Based on the above analysis, we illustrate the design motivation of TARSS-Net for RSS task one by one (see L144-L155), as well as detailed implementation methods in Sec. 3. Also, we further verify the superiority of TARSS-Net over the existing methods that consider temporal relation information.

We believe that the confusion you mentioned can be eliminated after readers carefully read the full paper, the Appendix and the code.

[W2. INSUFFICIENT EMPHASIS ON THE DESIGN MOTIVATION.] In Sec. 2, we have conducted a comprehensive discussion for existing temporal modeling paradigms, and based on the above analysis, we illustrate the design motivation of TARSS-Net for RSS one by one, including the design motivation of TH-TRE module. We believe that after you go back and read the first two sections of this paper again, your questions will be answered.

[W3. ISSUES WITH PAPER FORMATTING AND VISUALIZATION.] Sorry for the reading trouble caused by unreasonable formatting and visualization. Due to the space limit of paper submission, we had to make some typography which might make it uncomfortable to read. These will be corrected in next manuscript version, including the order of Fig. 3 and Fig. 4, more rigorously draw for elements in the figures, etc.

[W4. POTENTIAL ISSUES IN THE EXPERIMENTAL SECTION.] Due to space limitation, we show the experimental results that can best help to verify the performance of TARSS-Net in the most concise way. Due to the sparsity of radar targets, the dense computation of SA will inevitably introduce redundant computations on irrelevant information thus degrading the RSS performance. The performance of the VIT model is supplemented in Table 1 of attached PDF for this rebuttal. We also promise to add it to Table S2 in the Appendix of the revised manuscript.

Questions

[Q1. THE CORE INNOVATION OF THIS PAPER.] The core innovation is to propose an effective temporal modeling method specific to RSS tasks, i.e., the plug-and-play TRAM which combines the advantages of causality, end-to-end learnability, constant model parameters under arbitrary length input, and linear growth of computational complexity with the length of the sequence. These advantages cannot be satisfied in the same time when using other existing temporal modeling methods including Tranformers, 3DConv, RNNs and HMMs. For its significance, innovation and advantages, please read the first two Sections of the paper in details. In terms of causal dilated convolution (CDC), it definitly meets the parallel-computation, larger RF with fixed-size kernels and causal computing mechanism, however the dilation rate should be pre-defined, i.e., if the input length changes, the hyper-parameter, dilation rate should be changed accordingly before training. While TRAM does not require any adjustment when handeling different lengths of input. Morover, as far as we know, CDC is not in the 3D form which has the limitation for handeling temporal-spatial data such as radar RAD sequence. Hence, instead of talking about CDC let's dive into 3DConvs, which are more prefered by the researchers in RSS field.

[Q2. IS IT MEANINGFUL TO DISCUSS REAL-TIME PERFORMANCE FOR RSS?] Yes, it is very important to discuss the real-time capability of RSS. As a remote sensing device, radar is applied in many fileds, such as automatic driving, security warning and so on. Taking Ku-band drone surveillance radar as example, the PRT (pulse repetition time) is around 80us, and it has 128 coherent pulses in one CPI (one Range-Doppler frame), then the data rate for detection will be $\frac{1 \times 10^6}{80 \times 128} = 97.66~\text{FPS}$. This requires subsequent signal processing and detection/segmentation algorithms to match this data rate as much as possible. Hence, in order to accurately detect and stably track the target, real-time performance is one of the important indicators of RSS task, which has practical significance at the application level. Taking automatic driving as another example, the moving car needs real-time feedback of detection results in the surrounding environment, otherwise it will lead to unexpected consequences. Therefore, RSS needs to balance accuracy and efficiency.

Weaknesses

[W1. FEW INNOVATIONS.] Thank you for recognizing the rationality of our approach. However, beyond reasonability, the novelty of TARSS-Net is also guaranteed. We feel sorry that we failed to make you see the novelty, as well as the careful thought and extensive experimental validation that went into it. Therefore, we think it is important to reiterate the real value and advantages of TARRS-Net. As we covered in Sec. 2, if you want to exploit temporal information in radar data, then you basically have 4 options: HMMs, RNNs, 3DConvs and Transformers. However, when we decide to use them, we encounter some unavoidable problems: i) as shallow probabilistic models, HMMs are very limited in their ability to express primitives for each single time step, and they cannot be trained end-to-end (L86~L89); ii) RNNs inherit the causal computing mechanism of HMMs and can be deeply trained end-to-end, but they cannot fully enjoy the computational efficiency of parallelization (L90~L100); iii) 3DConv can be fully parallelized, but its calculation nature is non-causal and cannot be adaptively adjusted according to the length of the input sequence, i.e., to process longer sequences, the network needs go wider or deeper, which leads to the introduction of more training parameters (L102~L114); iv) the Transformer with SA as the core overcomes the problem of limited local receptive field when 3DConv deals with long sequences, and also overcomes the defect of RNNs that cannot be parallelized. However, its computational complexity will show square level growth with the length of the sequence, resulting in computational redundancy (L115~L127). Seeing these problems, we deeply realized that there is NO FREE LUNCH, we have to redesign temporal modeling method for radar data, and RSS model should not be stagnant on these methods, it should have better sequence modeling methods. To this end, we meticulously redesign the temporal modeling method and propose the design principles, resulting in TRAM, which combines the advantages of causality, end-to-end learnability, constant model parameters under arbitrary length input, and linear growth of computational complexity with the length of the sequence. Without careful design, we will not find a model in current methods that can achieve all of these advantages and still have the performance of SoTA. Therefore, we hope you to understand our efforts and recognize our work.

[W2. PERFORMANCE IN CROWDED SCENARIOS.] The concern you raised is indeed worth discussing. In real-measured radar data, target signatures often show extremely sparse characteristics (this does not refer to the physical size of the target, but to the characteristics of the target reflected in the radar signal)，so sparse and small target detection is a classic pain point which radar signal researcher dedicated to solving. So, if we understand your concern correctly, the RSS task does not suffer from the problem of model performance tends to degrade in crowded scenarios, which is one of the reasons why TARSS-Net emphasize the importance of learning and exploiting temporal relationships to enhance the representation of sparse target signature for RSS.

[W3. DIFFICULTY OF PERFORMANCE IMPROVEMENT ON RA-VIEW.] This is a good question that points out something unique to the RSS domain. The mismatch between the quality of RA-view data (for range and angle measurement) and RD-view data (for range and Doppler measurement) is a common phenomenon. Restricted by the hardware, most radars pay more attention to the ranging accuracy when designing, and the angular accuracy of the commonly used low-cost radar is hard to guarantee. Therefore, the difficulty in improving the segmentation accuracy of RA is more caused by the poor data quality, rather than the failure of the temporal modeling method. [35] notices this problem and modifies annotation quality of RA, resulting CARRADA-RAC dataset. Comparing Table 1 and Table 2, it can be seen that annotation correction can improve the performance of RA-view from 51.3% to 58.7 % (TARSS-Net_D). Therefore, it is reasonable to think that there is more room for the temporal modeling method to play a role after further improving the data quality.

[W4. NOT COMPREHENSIVE RELATED RSS WORKS.] Temporal modeling is not an emerging topic in other fields. However, effective and necessary modeling of temporal relationships has not been fully emphasized in RSS. Therefore, in order to encourage readers to revisit the temporal relation modeling from the perspective of radar signal processing and understand the motivation of TARSS-Net, we focus on the analysis of existing temporal modeling methods, the obstacles to their application in RSS domain, and the elaborate design of TARSSnet to face those obstacles. It is believed that Sec. 1 and 2 can bring somehow inspiration for readers in the NIPS community. It is worth noting that in SoTA method comparison (see Sec. 4.2), we have included as comprehensive as possible the existing excellent RSS works and made a brief analysis. Limited by the length of this paper, please visit the relevant index to download original paper for details of these works.

[W5. PLACEMENT OF ILLUSTRATIONS.] Thanks for your kind reminder and we are sorry for the reading trouble caused by unreasonable placement. Due to the space limit of paper submission, we had to make some typography which might make it uncomfortable to read. These will be corrected in next manuscript version, including the transposition of the order of Fig. 3 and Fig. 4, as far as possible to supplement the main text to reduce the redisplay in Appendix, etc.

Questions

[Q1. MORE DETAILED ANALYSIS OF THE COMPUTATIONAL COMPLEXITY.] There is real-time performance comparison in Sec. E.2 of Appendix and Table 1 of PDF for rebuttal, including model size, MACs, FPS and metrics.

Weaknesses

[W1. LIMITED TECNICAL CONTRIBUTION.] Sorry to have caused the reviewer such concerns. The core point of this paper is to redesign a better spatio-temporal modeling method for radar data from the perspective of temporal information utilization. Indeed, there are many works in other fields that discuss the utilization of spatio-temporal information. However, as we analyzed in L34~L44, there is still a large research gap in the related research of radar-oriented deep learning models: "In terms of temporal information utilization, the common practice is still 3DConv." Hence, we dedicated to exploring an advanced temporal modeling mechanism, whcih is urgently needed for modern radar (alleviating the problem of time-sensitive changes in radar data quality and learning better target representations). For the insights of RAD data handling, it is not the main focus of this paper, for two reasons: i) TMVA-Net [21] had provided a multi-view learning approach that well balances information utilization and efficient processing for RAD data; ii) conducting detection/segmentation on RAD data is not universal in radar systems, for some phased array radar system, it is more efficient to perform the detection directly on RD, e.g., KuRALS dataset used in this paper. In summary, this paper aims at the problem of temporal modeling which is more general in radar systems. From this perspective, this work summarizes the advantages and disadvantages of modern temporal modeling methods in L82~L127 and the problems that need special attention when processing radar data in L130~140. We will reiterate our core research objectives in the Introduction section to highlight our contributions.

[W2. MARGINAL PERFORMANCE INCREMENT.] Sorry not to surprise you, but this is actually not a small improvement for RSS tasks. As we mentioned in L20~L24, radar data is different from image data. It is susceptible to various interference and does not have semantic information, which make RSS more chanllenging. From the experimental part, it can be seen that the general semantic segmentation model of CV has not achieved good results on radar data (such as FCN, Unet). Then TMVA-Net [21], has improved the performance of RSS by about 2% (taking RD view as an example). The performance improvement of subsequent TransRadar [5], TransRSS [36] and PKCIn-Net [35] also increases by about 2%. To date, from FCN to TARSS-Net, RSS performance has improved from 66% to 75% on benchmark dataset, CARRADA. As you can see, RSS performance is thus advanced bit by bit. That said, the authentic and reproducible 2% performance improvement achieved by TARSS-Net is significant for the progress in RSS field.

[W3. LACK OF RUNTIME STATISTICS.] Due to page limitation, the runtime performance comparison is listed in Sec. E2 of Appendix. TARSS-Net_D takes 43ms to infer one RAD@5Frames, 9 ms for one RD@5Frames and 9 ms for RA@5Frames (tested on single Nvidia3090).

Questions

[Q1. RESULTS WITH 2 FRAMES.] Thanks for your suggestions to make our experiment more comprehensive and solid. We further tested TARSS-Net_D with 2 input frames on RD view CARRADA: mDice 73.7%, mIoU 62.0%, Precision 71.3%, Recall 76.5%. It is supplymented in Figure 1 of attached PDF for this rebuttal.

[Q2. ANALYSIS OF PERFORMANCE DECREASING AFTER 6 FRAMES AND INCREASING AGAIN FROM 9 FRAMES.] This is a particularly valuable question. We did do some analysis of the trend of the curve shown in Fig. 5, and try to summarize some general conclusions that can guide TARSS-Net usage. Different operating conditions of radar affect the quality of each data frame, so in one RSS dataset, it is hard to generalize how many historical frames are closely related to the target one, or which historical frames are helpful to the current frame (poor quality history frame will introduce a lot of irrelevant noise and lead to performance degradation). Not to mention different RSS datasets. All networks dealing with temporal-sturctured radar data face the problem that they cannot calculate the correspondence between the number of input frames and network performance. TARSS-Net has its own advantages to help ease the choice of input frame length. With the design of TRIC layer, TARSS-Net could accept input with adjustable time length while keeping the number of parameters constant. When using it, readers can try and choose the number of input frames according to constraints of inference performance of the hardware (TFLOPS) without worrying about model size.

[Q3. Feature visualization.] Please see Figure 2 in attached PDF for this rebuttal.

We have provided detailed responses to all of your questions. Hope to get your approval on the reply and update the rating. We also look forward to more in-depth communication and discussion with you! Thank you again!

Dear Reviewers,

Sorry to bother you, but this is the last reminder from the authors. We sincerely invite all reviewers to review and evaluate our responses. We thank you for your hard work in the early stage and hope to get your short attention again.

Since there are only three hours left for discussion, we attach great importance to this submission and would like to buy you a little time at the end.

Warm regards,

9831 Authors