Radar Spectra-language Model for Automotive Scene Parsing

Thank you for your comments and suggestions! We address the issues raised below

R1:S1 The presented work is one of the first to train a radar-language model for autonomous driving scene understanding.

It is the first to train a radar spectra – language model, and the first to make use of the shared language model embedding space to shed light on the semantic content of radar spectra.

Radar spectra is a unique form of measurement – it captures the spatial reflection intensity of the electomagnetic signal and thus the geometric structure of the environment. But it also captures the reflection intensity along the Doppler axis, i.e. it is (radial) motion-aware, and the reflection intensity depends on the material properties, as well as geometry. It is clear that radar spectra carry richer information than the widely used sparse radar reflection points, but it is not quite known what kind of information can be extracted. The current work aims to provide a way to shed light on this.

R4:W1 Please consider comparing your proposed model against some existing adapted language models for driving scene understanding.

A review of VLMs for automotive scene understanding was added to the related work section (edited part highlighted in blue). All methods are quite new - concurrent to our work. None of the reviewed methods is quite applicable to our setting of retrieval – where data is scarce and semantic ground truth is absent (and off-the-shelf VLM performance is unsatisfactory).

R4:W2 Please consider directly comparing your proposed model against some object detection and segmentation models to verify the superiority of your proposed model.

The comparison of our proposed model to an existing radar object detection and segmentation model is included in Table 3. The baseline model in Table 3 is FFT-RadNet of Rebut et al. (2022), the SOTA for the RADIal dataset, we clarified this in the table and the corresponding section 4.3 (updated portions of the manuscript are highlighted in blue). Following this comment we added to the comparison SotA results from two additional papers on the same dataset.

R4:W3 Most of the components from the proposed framework are from existing works. It is hard to find any novel technical designs in the presented framework. Please better clarify the technical novelty and theoretical contributions of the presented work.

Our paper introduces a novel task of extraction of semantic information from radar spectra – an otherwise generally non human - interpretable modality, and introduces an approach to tackle it. Both of the above are the main novelties of the paper. It is fortunate, that the encoder of an existing object detection and segmentation network (FFT-RadNet Rebut et al. (2022)) can be used as radar encoder of the radar spectra language model. This way, not another specific architecture has to be designed for one specific use case.

In addition, we describe a novel process of generating captions for and fine-tuning an image-language model to extract general semantic information from AD scenes.

Finally, to validate our radar-spectra model we modify an object detection model acting on spectra by injecting the embedding vector from our spectra encoder, and demonstrate it improves SotA on the RADIal dataset.

The goal of the work is to explore the semantic content of radar spectra, which is otherwise very difficult.

In the updated paper version, we explain our contributinos in a clearer manner (highlighted by blue text color). We will try to clarify this further in the final version.

R4:W4 Would you consider giving an overview of your proposed framework at the beginning of the methodology section, which can help the readers better understand the work?

Done. Overview provided at the begining of the "Proposed Approach" Section 3.

R4:W5 The computation efficiency should be discussed, which is critical in automotive scene understanding.

We agree that computational efficiency is an important property for models running on an embedded device. Nevertheless, the main focus of the paper is to a) get a better understanding of the content of radar spectra and b) to investigate the benefits of the learnt features for downstream tasks. The focus is not on computational efficiency and the method is not intended to be run on an edge device, but rather is a research tool.

In other words, the current work aims to provide a tool to examine the semantic content of radar spectra, rather than provide a perception algorithm for an autonomous vehicle. For this reason, computational efficiency is not a major consideration.

R3:W1 The author seems to lack paper writing skills. All the figures are unaesthetic bitmaps with low resolution and some of the figures are not necessary. For Figure 4a, it is better to use formulation instead of python code to describe the loss functions. For Figure 4b, such a simple architecture may be put in the supplement material.

Figures were re-exported in vector format.

Figure 4 has been moved to the appendix and re-numbered to Figure 6. Formulation of loss functions was added in Section A.2.

The reason we used python code was to be consistent with the CLIP paper Radford et al. 2021, which defined its InfoNCE loss with python code.

R3:W2 Changing the position encoding without finetuning may cause performance drop, and splitting the image may break some objects on the edge. A better and more common way is to pad black pixels to the top and bottom of the image.

We always fine-tuned the model when using each of the adaptation methods we reported ( (1) positional encoding interpolation, (2) multiple square crops).

When creating crops we made sure there's an overlap, hopefully mitigating effects like mentioned - of breaking an object on the edge, i.e. an object would usually appear in its entirety at least in one of the crops.

Following this comment we tried using zero-padding, which worsened results (average weighted accuracy in the experiment did not exceed 0.5).

We believe that the reason is that images in the RADIal dataset have aspect ratio of nearly 1:2, while the network input is 244x244 square. In this case (resize and) padding results in half the input image being black, with effective image height just 122 pixels, and many details are lost – resulting in the observed performance hit.

R3:W3 For the detection and segmentation downstream tasks, it is better to show some cases that the pretrained model helps improve the performance, not just numbers.

We added a new appendix section A.6 with visualization of object detection and segmentation, and a brief discussion.

R3:W4 For autonomous driving tasks, the localization abilitiy is more important than the classification. Could you provide some visualizations such as attention map or GradCAM to see if the retrieved objects are corresponding to the right location?

We added a new appendix section A.4 with visualization of VLM activations using GradCAM before and after fine-tunning.

Other issues:

• For Equation 4, a period should be added in the end of the formula.
• All the quotation marks are single quotes. Please use backquote in front of the quoted phrases.
• Some of the RADIal is mistaken by RADiaL.

All done. Thanks for pointing out!

Thank you for the valuable feedback! We would like to answer your questions:

R2:W1 No discussion on what is still hard to do or not reliable. Also analysis of the varying the difficulty of the input scenes would help answer the previous question.

A discussion section was incorporated (page 10). The changes in the updated paper version are highlighted with blue text color.

R2:Q1 What self-driving related take would Radar-spectra be able to do well, while other modality cannot or struggle with?

Radar has several advantages compared to other modalities, e.g., it is robust to difficult weather conditions like rain or sun glare, and it can directly measure the relative radial velocity of other objects. Moreover it can detect objects at large distances, depending on the sensor for example up to about 350m. Nevertheless, the focus of the paper is not a comparison of radar to other modalities. We investigate the semantic content of radar spectra, which is otherwise difficult to interpret, unlike other modalities.

R2:Q2 Typos: Pg. 3, "Prompt Generation" section: (e.g. a photo of a {}) -- {} symbol should be replaced.

Done! Thank you for pointing this out!

We thank you for your valuable comments and suggestions! Below we address the issues raised.

R1:W1 The framework seems to be a simple combination of existing methods. I didn’t see the specific design for the radar spectra language model.

Our paper introduces a novel task of extraction of semantic information from radar spectra – an otherwise generally non human - interpretable modality, and introduces an approach to tackle it. Both of the above are the main novelties of the paper.

In addition, we describe a novel process of generating captions for and fine-tuning an image-language model to extract general semantic information from AD scenes.

Finally, to validate our radar-spectra model we modify an object detection model acting on spectra by injecting the embedding vector from our spectra encoder, and demonstrate it improves SotA on the RADIal dataset.

R1:W2 The experiment of detection is not compared with SOTA methods such as RODNet.

and

R1 :W3 What is [20] in Table 3?

Thank you for pointing this out! The reference in table 3 is the FFT-RadNet model from Rebut et al. (2022) - it has been fixed in the updated paper version. FFT-RadNet is the SOTA model for the RADIal dataset, to which we compare our proposed methods. We consider the RADIal dataset for the evaluation of the proposed radar spectra language model. Unfortunatelly, RODNet cannot be applied to this dataset directly, since RODNet is designed to work on a sequence of complex valued range-azimuth-spectra, the data provided in the CRUW dataset. In contrast, the RADIal dataset contains range-Doppler spectra, where the azimuth dimension has not been processed yet.

Following this comment we've added results of two more SotA methods for RADIal to the comparison. We've extended Sec. 4.3, in particular the ``Results" paragraph to accomodate for the changes. Modified portions of the manuscript are highlighted in blue.

R1:W4 If the description includes multiple object information, how do you align the text information with the corresponding object?

In this work, we formulate queries as descriptions on scene level, we do not align text information with a specific object explicitly. When generating captions, we have the entire information of the scene. In conclusion – the alignment problem does not arise, at least for the setting described in the paper.

R1:Q1 If the description is not accurate will the information mislead the model?

In general, yes - imprecise descriptions may have a negative effect of training. For example, using inaccurate descriptions for highway scenes, e.g. "there are 3 cars in the street" (as opposed to "on the highway"), decreased the quality of the scene classification. That's why we don't use GPT augmentations of captions, as it sometimes adds information that is not relevant to the image like "it's a sunny day, car goes down the street" for a picture with cloudy weather. Meanwhile, we added augmentations that are supposed not to violate caption-image aligning, e.g. "it's a photo of the environment with a car" and "there is a car".