CEDNet: A Cascade Encoder-Decoder Network for Dense Prediction

1. The novelty of the block design is limited. The CED block and LR-CED block are similar to the block used in ConvNext. It looks like a tiny modification to the existing method. The long-range interactions are not really justified or analyzed.
2. The motivation for the cascade design is not very clearly explained. The authors claim it allows high-level features to guide low-level features, but do not provide analysis into why proposed cascading enables this better than other approaches ($i.e.,$ delayed fusion).

1. In Table 1, the flops for the proposed methods may have one typo. 255M->255G?

2. The CED block used in the proposed methods is similar to the ConvNeXt block [1]. Please further describe the difference between these two modules and provide a deeper analysis.

3. Table 1 shows the FPN-style encoder-decoder slightly outperformed the models built on the UNet-style and Hourglass-style encoder-decoder on object detection for the baseline ConvNeXt-T. Please further compare the proposed three encoder-decoder architectures with the larger baselines ConvNeXt-S and ConvNeXt-B.

4. For the pre-training, besides the last decoder, the author claims that the parameters of the proposed encoder-decoder method CEDNet have been propagated by the Imagenet-1K datasets. For the proposed cascaded encoder-decoder architectures, if there exist some learned layers in the decoder part, these layers have also been pre-trained. For the downstream dense tasks, as shown in Table 2 and Table 3, the compared method ConvNeXt-T only adopts the classical FPN as the default fusion module. Considering ImageNet-1K has more training images than the COCO and ADE20K, it seems the massive pretraining of the decoder could benefit the performance of the final dense prediction tasks.

[1]. A convnet for the 2020s，CVPR2022.

There are some key concerns:

1.Technical contributions

In fact, this work is largely based on previous works, such as cascade structure, Hourglass,U-Net and FPN, etc. The key differences are the cascade structure. However, in my knowledge, the proposed methods have appeared in previous works, such as Stacked U-Net, Cascade U-Net/FPN. There are only small differences in the block design. The macro idea is the same. There are no essential differences in theory. As for the CED block, the authors just use typical residual modules with recent tricky improvement. I agree that we need more powerful multi-scale feature fusion methods for dense prediction task. However, the methods in this work are not a game-changer for this topic. There are too many existing modules and techniques.

2.Insufficeint experiments

First, there are no full comparisions with recent methods. For the object detection task, the authors only compare with out-of-date methods in Tab.1 (Only one method after 2022). Should the authors provide more reuslts with recent works? I suggest the authors add more comparisions with recent methods. As far as I know, there are many methods perform better than the proposed. In addition, I wonder why the CEDNet-NeXt-T (Hourglass-style/U-Net-style/FPN-style) have the same Param. and FLOPs. I think it is impossible. Second, in Tab.4 and 6, the authors keep the parameter number to verify the effects. I think the direct cascade the structure is more reasonable. How about the results? In fact, if the stage one is very powerful, other stages are not needed. The gains may be too small. Third, there are no enough examples to show the visual results of three tasks. In fact, the authors only present the results in tables Please list the visual results. There are no visual examples for the failure cases. Finally, the LR CED has a very big effect for the performance as shown in Tab.8. I think it weakens the contributions of other modules when compared with the results in Tab.2.

3.Unclear details

There are some unclear details. For examples, how to get 1/8 feature maps with two sequential 3×3 convolutional layers (each with a stride of 2)? Should it be 1/4? What is the effect of using differen dilation rate in LR CED?

• After attentively reading the paper, a conflicting impression emerged. Even if there are noticeable improvements over benchmarks, there is yet little concrete proof to support the claim that late fusion is less beneficial, which seriouly undermines the credibility of the motivation and the soundness the whole work. If the authors can locate some concrete evidence to pinpoint the issue, such as a feature map visualization (only a hint instead of remanding), it will greatly enhance the credibility. One such visualization is disappointingly only available in the supplemental file and is not included in the main paper.
• It appears that the authors' understanding of low- and high-level features is flawed. Low-level feature is not identical to high-resolution feature and high-level feature is not the low-resolution one. In CV field, corners, edges, textures, and colors are examples of low-level features, whereas items, scenes, and behaviors are examples of high-level features. High-level features typically originate from deeper layers, while low-level features are typically found in earlier layers. In the conventional pyramid structure of modern neural networks, high-level feature happens to be the low-resolution one. Thus, the key claim is questionable that the CEDNet design can incorporate high-level features from early stages to guide low-level feature learning in subsequent stages.
• Stacked encoder-decoder is not something fresh, e.g., CascadePSP [1], SDN [2].

[1] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement. CVPR 2020.

[2] Stacked Deconvolutional Network for Semantic Segmentation. TIP 2019.