Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation

Thank you Reviewer zhiy for your time to thoroughly evaluate our paper and your valuable feedback. It has greatly raised the quality of our manuscript. Please feel free to add any further questions based on our responses.

---

• Response to Weakness 1. M2F's encoder system (or pixel decoder) consists of two parts- a transformer encoder that produces enhanced multi-scale tokens and a set of convolutional layers that produce the per-pixel embedding map. PRO-SCALE and the LPE module play crucial roles in optimizing computational efficiency of these components:

  • In original M2F, multi-scale features from the backbone are processed through transformer layers in the encoder, which can be computationally expensive due to the global attention mechanism. PRO-SCALE significantly reduces this computational cost by introducing progressive token scaling, which adaptively reduces the number of tokens (and hence computations) at each stage while preserving key information.
  • The pixel embedding map is generated using convolutional layers, which enhance local spatial details. However, the convolutional layers require processing an enormous number of tokens, leading to extremely high computational demands. With a simple structure of the LPE module, we show that by weakening the inductive bias due to the convolutional layers, the model still maintains high-quality results without taking on the computational burden.

  This discussion has been added in the manuscript.

---

• Response to Weakness 2. The stride is set to 1. The kernel size is set to 3 (provided in L249 of the submission version). These are the default parameters of the Maxpool2D operation which we didn’t change. We have added these details to the paper.

---

• Response to Weakness 3. Our aim was to ensure the main contribution of the paper— progressively expanding tokens along the encoder depth to address the computation redundancy—remains clear and central to the reader. To avoid potential confusion, we did not include the token recalibration operation in the main figure but explicitly mentioned it in L233-236 and provided a detailed illustration in Figure 7 of the Appendix. We have added a note for the reader to refer to this figure for a complete visualization.

---

• Response to Weakness 4. In the submission version, we mentioned in L790 that the TRC unit only costs 0.006 GFLOPs (computed on the COCO).

---

• Response to Weakness 5. Thank you for the suggestion. We added FPS analysis for PRO-SCALE and other methods in Tab. 15 and Tab. 16.

---

• Response to Weakness 6. We agree with the reviewer that with larger backbones, such as SWIN-L, backbone computations dominate. To address this, one can switch to a smaller backbone like SWIN-T to reduce backbone costs and incorporate PRO-SCALE to further optimize encoder computations, making the overall model more efficient.

---

• Response to Weakness 7. We have updated the abstract to clarify that the GFLOPs reduction is for the encoder.

---

• Response to Weakness 8. We have updated these tables to show the GFLOPs of the entire model.

---

• Response to Weakness 9. ReMaX is a training algorithm designed to train small (or efficient models) and improve their ability to create better panoptic masks without any additional parameters. It doesn’t increase the efficiency but the performance. The GFLOPs of ReMaX trained models shown in Tab 1 and 2 have been updated now.

---

• Response to Weakness 10. We agree with the reviewer that SL has lesser degradation than MoBy when compared to M2F. However, we wanted to say that integrating PRO-SCALE with the MoBY pre-trained backbone results in better performance compared to using SL backbone weights. We have modified the statement to reflect this.

---

• Response to Weakness 11. Thank you so much for pointing these out. We have corrected all the typos in the revised paper.

---

• Response to Weakness 12. Thank you. We have updated the caption of Tab. 3.

---

• Response to Weakness 13. Thank you. We have updated all the tables to make the decimals consistent.

---

• Response to Question 1. Small-resolution features encode compact, highly semantic information, whereas high-resolution features primarily capture fine-grained local details. In Appendix B, we illustrate that the abundance of high-resolution features often results in redundant information. Updating these features less frequently has minimal impact on performance while significantly reducing the computational cost associated with the attention mechanism. Hence, PRO-SCALE progressively adds the higher resolution features at later layers, thereby prioritizing semantic refinement in most layers. This reduces the number of query tokens, enabling a more computationally efficient multi-scale encoder.

---

Thank you Reviewer zhiy for your time and raising further questions. Please feel free to add any further questions based on our below responses.

---

• Response to comment 1 "difference between LPE module and the existing method".

  • The main focus of LPE module is not proposing a more efficient alternative to convolutions, but rather demonstrating that the original pixel embedding layer can be replaced with LPE—a lightweight, non-convolutional module—while maintaining competitive accuracy. This finding is significant because it advances the segmentation model architecture by identifying and addressing inefficiencies in the original design.

  • At the architecture level, the differences in the embedding layer are the following:

    • The original M2F ‘s embedding layer has the following structure: lateral_conv (feature_in_channel, conv_dim, kernel_size=1) -> output_conv (conv_dim, conv_dim, kernel_size=3). In this, the largest computations come from the output_conv (conv_dim, conv_dim, kernel_size=3) layer.
    • Our embedding layer has the following structure: lateral_conv (feature_in_channel, conv_dim, kernel_size=1) -> LPE (conv_dim, conv_dim, kernel_size=3)

    Here, lateral_conv and output_conv are detectron2's Conv2D [A] operations. Following M2F, conv_dim is set to 256 and feature_in_channel is the number of channels of highest resolution feature map $s_1$. Both the convolutional operations and LPE modules are followed by normalization and non-linearity.

  [A] https://detectron2.readthedocs.io/en/latest/modules/layers.html#detectron2.layers.Conv2d

---

• Response to further query on FPS.
  • FPS for PRO-SCALE (2,2,2) and PRO-SCALE (3,3,3). As per reviewer's request, the FPS of PRO-SCALE (2,2,2) and PRO-SCALE (3,3,3) are provided in the table below (with SWIN-T backbone on Cityscapes). | Method | PQ | FPS | | ------------ | -------- | ----- | | Lite-M2F | 62.29 | 6.01 | | Ours (1,1,1) | 60.58 | 6.47 | | Ours (2,2,2) | 62.37 | 6.07 | | Ours (3,3,3) | 63.06 | 5.71 |
  • PRO-SCALE vs Lite-M2F benefits. PRO-SCALE is better than Lite-M2F, as can be seen in the following results.
    • As seen in the table above, PRO-SCALE clearly betters Lite-M2F both in performance and FPS (eg. configuration (2,2,2) provides better results with similar FPS).
    • On a larger and diverse dataset like COCO, PRO-SCALE outperforms Lite-M2F significantly. For instance as shown in Tab. 1 with SWIN-T backbone, Lite-M2F provides a PQ of 52.70 with 31.81% encoder GFLOPs reduction whereas PRO-SCALE (3,3,3) results in a PQ of 52.82 with 51.99% encoder GFLOPs reduction.
    • PRO-SCALE shows better performance than Lite-M2F performance both in segmentation and detection tasks as shown in Tab. 8 with lesser compute cost.
  • FPS results in Tab. 3. Based on the reviewer’s suggestion, we updated Tab. 3 with FPS values (last column) in the revised manuscript.

---

• Point regarding larger backbones. We appreciate the reviewer’s view-point here. We have the following comments.

  • Clarification on effectiveness of PRO-SCALE on large backbones. PRO-SCALE is effective for M2F models with both large and small backbones. Regardless of the backbone size, it significantly reduces the encoder cost without causing a notable performance drop.
    • With larger backbones like Swin-L, PRO-SCALE has a huge impact on the encoder's efficiency (50.87% reduction in table below). However, the total GFLOPs may not decrease dramatically (7.48% reduction) because the backbone's compute cost still dominates the overall model cost.
    • In contrast, with smaller backbones like Swin-T, the encoder accounts for a larger proportion of the total compute budget. Hence, PRO-SCALE's impact on the encoder (51.96% reduction in table below) reduces the overall model compute cost significantly (27.09% reduction).
      To demonstrate the above two points quantitatively, we summarized the results from Tab. 2 and Tab. 7 below. | Backbone | Original PQ | With PRO-SCALE PQ | Encoder Cost Reduction | Total Cost Reduction | |-------------------------|-------------|--------------------|-------------------------|-----------------------| | Smaller Backbone (SWIN-T) | 64.00 | 63.06 | 51.96% | 27.09% | | Larger Backbone (SWIN-L) | 66.60 | 65.93 | 50.87% | 7.48% |
  • Response to If one wants to achieve the best possible segmentation performance, then opting for a small backbone instead of a larger one is typically not an option, as this limits the performance. Please note that our core focus is to balance efficiency and performance in cost constrained settings, where smaller backbone excel (and with small backbones, encoder cost is the highest for the whole M2F model). Of course, larger backbones based M2F models excel in high segmentation quality, however, they fall short in resource-constraint scenarios. We summarize these two points in the table below. | Aspect | Small Backbones based M2F | Large Backbone based M2F | |-------------------------------|-------------------------------------------------------------------------------------------------------|------------------------------------------------------------| | Performance-Accuracy Tradeoff | - Balanced accuracy and efficiency. | - Achieves high segmentation accuracy, but poor tradeoff. | | Deployability | - Scales well across devices with low memory and compute needs. | - Less suited for setups prioritizing compute cost. |

  To conclude, if we require to use M2F in a resource-constrained environment, opt for a smaller backbone and use PRO-SCALE for the encoder to achieve optimal performance.

---

• Comment on MoBY vs SL weights. Thank you for the suggestion and we agree with the reviewer. We have modified the statement in the paper with “integrating PRO-SCALE with the MoBY pre-trained backbone results in better (overall) performance compared to using SL backbone weights.”

---

• Response to question Why does the presence of redundant information mean that these features can be updated less frequently with minimal impact on performance?. Here's some additional description.

  • As shown in Fig. 9, low-resolution tokens ($s_4$, $s_3$​) demonstrate less redundancy, while high-resolution tokens ($s_2$​) shows more. The original M2F's encoder refines multi-scale features, concatenated along the token dimension to form $S=[s_4,s_3,s_2]$, by using deformable transformer attention [B]. Deformable transformer attention mechanism ensures that each token attends only to a small, learned set of keypoints across scales instead of the entire feature map. However, refining $S$ across all six encoder layers incurs significant computational cost.
  • Now, since deformable attention requires only a few keypoints from each scale and $s_2$ is highly redundant, prioritizing frequent updates to less redundant features ($s_4, s_3$​ which contain compact information and rich semantics for most details of the scene [C]) maintains performance similar to the original model but drastically reduces computational cost.

  Hence, in PRO-SCALE, $s_2$ is only introduced in layer set $p_3$ whereas $s_4$ is updated all layer sets $p_1$, $p_2$, and $p_3$.

  [B] Deformable DETR: Deformable Transformers for End-to-End Object Detection, ICLR 2021
  [C] Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR, CVPR 2023

---

Hi Reviewer zhiy,

Thank you for extensively engaging in discussions during this rebuttal period. We understand the reviewer wants to retain their score. We would like to highlight the following.

Note: Our work targets reducing computational complexity (FLOPs), not FPS, a hardware-dependent metric. FPS reduction is orthogonal to our focus, and we will address this in our future works. The reviewer have themselves stated that our work is valuable because it critically examines an inefficient component of a frequently used model, and finds that it can be made considerably more efficient while keeping the performance roughly the same.

---

• Performance w.r.t. Lite-M2F.

  • Measuring using FLOPs: The reviewer agrees with us that PRO-SCALE is better than Lite-M2F when considering GFLOPs (PRO-SCALE provides 51.99% reduction while Lite-M2F provides 31.81% reduction) while providing higher quality PQ (PRO-SCALE (3,3,3) PQ of 52.82 vs Lite-M2F PQ of 52.70). Clearly, PRO-SCALE is more preferable.
  • Measuring using FPS: Based on the reviewer's request, we did an FPS analysis. With the point that we did not aim to optimize for FPS efficiency in our work,
    • PRO-SCALE is again better than Lite-M2F even when considering FPS (PRO-SCALE provides 6.07 FPS while Lite-M2F provides 6.01 FPS) while providing higher quality PQ.
    • The gains are agreeably minor, but again, PRO-SCALE is still more preferable.

  Conclusion: Given the above two points, it is respectfully incorrect to say that "As a result, there is no clear benefit of PRO-SCALE over existing method Lite-M2F", when the quantitative evidence suggests otherwise. This is further in contradiction to our range of ablation studies of application of PRO-SCALE involving different types of backbone (Tab. 5), different types of weight initialization of the backbones (Fig. 5), different scales of backbones (Tab. 7), and different types of tasks that go beyond the universal segmentation (detection in Tab. 8, open-vocab segmentation in Tab. 10 and multi-task prediction in Tab. 11).

---

• Contribution of LPE.

  • What we claim: The main focus of LPE module is not proposing a more efficient alternative to convolutions, but rather demonstrating that the original pixel embedding layer can be replaced with LPE—a lightweight, non-convolutional module—while maintaining competitive accuracy.
  • How we validated: We showed using GFLOPs that it reduces the complexity of the model further by 25% decrease. The reviewer rightly points out that the efficiency impact in terms of GFLOPs is large. Even with FPS (again not our focus), this module provides a better speed.

  Conclusion: Performance maintenance is evaluated from the perspective of the entire model, including PRO-SCALE. LPE focuses solely on reducing computational inefficiencies in design, and it effectively and objectively achieves this goal (25 % FLOP reduction and 2% better speed).

---

We again appreciate and thank the reviewer dedicating their valuable time in discussing with us. We hope the reviewer supports our work. We will focus on improving FPS improvement in our future works.

Hi Reviewer zhiy,

We appreciate your discussion on the significance of FLOPs reduction in connection to inference speed, and we would like to address the points you have raised, especially "if such a speedup is not achieved, what is the point of having a low number of FLOPs?". We start by acknowledging the validity of your request for inference speed comparisons. However,

• It is not fully accurate to suggest that reducing FLOPs has no value unless it translates directly to faster inference speed, especially on high-performance GPUs. While our experimental results indicate that FPS improvements on such GPUs may be limited, lowering the number of FLOPs is fundamentally important for several reasons even if immediate FPS gains are not achieved on "modern hardware":

  • Energy Efficiency: FLOPs correlate with energy use [D]. Reducing FLOPs enables more energy-efficient models, essential for large-scale and low-power deployments.
    • The reviewer may have overlooked the additional axis of efficiency that GFLOPs reduction offers beyond FPS.
  • Deployment on Edge Devices and Compatibility with Hardware Accelerators: These devices cannot use modern high-end GPUs, and hence require fewer FLOPs to ensure feasibility. Fewer FLOPs improve compatibility with hardware like mobile devices and specialized processors, enhancing deployment on constrained systems.
    • The reviewer appears to focus solely on high-end GPUs, but this overlooks the broader applicability of a model designed to run effectively on lower-end or specialized hardware, where FLOP reductions can play a crucial role.

  Conclusion: PRO-SCALE's significantly lower FLOPs compared to baselines is a strong benefit beyond high-end GPU scenarios such as energy efficiency and feasibility with low-compute devices, and we show this with strong quantitative evidence via GFLOPs reduction.

We appreciate the opportunity to engage in this discussion and thank you again for highlighting these points. Your feedback has allowed us to improve our paper (especially from the first feedback) and provide a more thorough understanding of our contributions.

[D] Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning Sustainable Computing: Informatics and Systems, 38, 100857, 2023.

Thank you Reviewer 1znQ for your time to evaluate our paper and your valuable support. Please feel free to add any further questions based on our responses.

---

• Comment on the abstract. We have updated the abstract to clarify that the GFLOPs reduction is for the encoder.

---

• Figure to show the FLOPs vs performance trade-off. We have added Fig.10 in the appendix to show the GFLOPs vs performance trade-off.

---

• Comment on impact of PRO-SCALE and LPE module.
  • We note that M2F calls the whole encoder system as “pixel decoder”. The pixel decoder provides two outputs to the next stage (i.e. the decoder): refined multi-scale features (using the transformer encoder) and the pixel embedding map (using the convolutional layers). Hence, we follow their system to design PRO-SCALE and the LPE module within this encoder system.
    • an example: In Tab. 3, it can be observed that if we replace the original M2F's encoder system with PRO-SCALE's (1, 1, 1) configuration (no LPE), the cost goes from 281 GFLOPs to 132 GFLOPs (reduction of 149 GFLOPS). If we further replace the pixel embedding layer with LPE module, the total cost goes from 132 GFLOPs to 73.49 GFLOPs (reduction of 58.51 GFLOPS). Clearly, it indicates that PRO-SCALE is extremely important for the reduction of computational complexity of the entire system as it results in more GFLOPs reduction.
    • Also note that, the main focus of LPE is not proposing a more efficient alternative to convolutions, but rather demonstrating that the original pixel embedding layer can be replaced with LPE—a lightweight, non-convolutional module—while maintaining competitive accuracy. This finding is significant because it advances the segmentation model architecture by identifying and addressing inefficiencies in the original design.
  • To avoid potential confusion on the main contribution of our paper, we did not include the token recalibration operation in the main manuscript due to the minor performance difference.

---

• Question on using depth-wise 3x3 convolutions in LPE. Our main focus is to show that the pixel encoder in the M2F architecture can be greatly lightened, rather than improving the efficiency of the convolution itself. While the reviewer’s suggestion of using depth-wise 3x3 convolutions aligns with the need to reduce computations, it would still introduce a larger computational burden compared to our proposed LPE module.

---

Thank you Reviewer g4Mz for dedicating time to evaluate our paper and your valuable support. Should there be any additional questions or feedback regarding our responses, please do not hesitate to raise further queries.

---

• FPS comparison with Lite-M2F and RT-M2F. Based on the reviewer's suggestions, we analyzed the FPS of Lite-M2F and RT-M2F against PRO-SCALE in the table below. PRO-SCALE achieves an optimal/better balance by delivering a high PQ score alongside strong FPS performance compared to other methods, ensuring an effective trade-off between quality and speed. PRO-SCALE has the distinct advantage to adjust its $(p_1, p_2, p_3)$ configuration to adapt to user's FPS vs performance requirements. | Method | Performance (PQ) | FPS | | ------------ | ----------------------------------- | ----- | | Lite-M2F | 62.29 | 6.01 | | RT-M2F | 59.73 | 6.59 | | Ours (1,1,1) | 60.58 | 6.47 | | Ours (2,2,2) | 62.37 | 6.07 | | Ours (3,3,3) | 63.06 | 5.71 |

---

• Comparison to ReMaX. ReMaX is a training algorithm designed to improve the panoptic mask quality of small or efficient models without introducing additional learnable parameters. While it enhances performance, it does not improve efficiency. As noted in Appendix C of the ReMaX paper, "the improvement of ReMaX gets saturated when the (model parameter) numbers become high." For instance, Table 11 in the ReMaX paper shows only a 0.3-0.6% PQ improvement on the kMaX-DeepLab baseline when applied to ConvNeXt-T and ConvNeXt-S backbones. In contrast, our method focuses on reducing encoder computations across all model sizes, providing a more consistent trade-off between efficiency and performance.

---

• Additional visualization comparisons. Based on the reviewer's suggestion, we added additional visualizations in the Appendix for Lite-M2F and PEM in Fig. 11 of the revised manuscript.

---

Thank you Reviewer GmJD for taking the time to review our paper and for your valuable support. Please feel free to raise further questions based on our responses, if any.

---

• Difference w.r.t. traditional fully convolutional network (FCN) style upsampling methods. The LPE module differs from traditional FCN-style upsampling methods primarily in its computational efficiency and design philosophy. Traditional FCNs use fully convolutional layers for pixel embedding, which require processing large input sizes, leading to significant GFLOPs and high computational costs. In contrast, the LPE module avoids these costly learnable layers, relying instead on lightweight operations to achieve similar functionality. By leveraging simpler mechanisms for pixel embedding, the LPE module enhances local detail and provides high-quality masks without adding computational overhead to the encoder. This design ensures a better balance between accuracy and efficiency, particularly for tasks requiring pixel-level detail.

---

• Using different number of layers for each split. We assume that the reviewer may be suggesting the use of a different number of layers for each split. The reviewer is correct on the conclusion that the computational costs in $p_1$ and $p_2$ are lower compared to $p_3$ (but since M2F uses deformable attention for the encoder layers, it is linear). As shown in Tab. 6 of the manuscript, we can definitely use a different number of layers for each split. Setting the ($p_1$, $p_2$, $p_3$) configuration values are independent of each other. The advantage is: different ($p_1$, $p_2$, $p_3$) configurations can allow multiple options for the encoder for the same computational budget.

---