We highly appreciate your valuable comments, especially for your attention to the details in our paper.

Response to Weakness #1. We conduct more experiment and provide the new experimental results on Cityscapes and Pascal Context (in Common Response). Due to time limitation, we did not provide ViT-Large based DEPICT on these two datasets (at this moment), but we will get it done soon (maybe during the discussion or at least in the final version). Based on ViT-Small, we show that our DEPICT outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context. Regarding the GFLOPs metric, our decoder requires remarkably fewer GFLOPs as it is fully attention based method and utilizes the MSSA block. The GFLOPs of our decoder is 2/3/4/10 based on ViT-Tiny/Small/Base/Large, whereas the segmenter's is 2/6/22/37.

Response to Weakness #2. The conclusion of line 144 is justified by that maximizing a scalar $x$ is equivalent to maximizing $\log(1+\alpha x)$ where $0 \leq x \leq 1$, $\alpha > 0$. The identity matrix of equation (3) degenerate to scalar 1, and the identity matrix of equation (4) has a shape of $N \times N$, where $N$ is the sequence length. Despite the dimensions are inconsistent, equation (3) is strictly equivalent to equation (4), by Lemma A.2 of [1]. For line 412, it is indeed a mistake, which should be corrected as "being equivalent to minimizing $|\beta|$". Thanks for pointing it! Fortunately, this mistake doesn't affect the correctness of the following equation (37). We are willing to checking or refining our proofs. For example, we have proposed much more concise and intuitive proofs from the perspective of low-rank approximation (see Response to Question#2).

Response to Weakness #3. It is our fault for providing no explanation about the font color. The red color means that the parameters are somewhat constrained as we constrained that $\text{heads}*\text{dim}$_$\text{head}=C$. However, as explained in our Common Response, we now remove this constraint, as it only holds for a single image rather than for the whole dataset.

Response to Weakness #4. This is a very insightful concern. A model for semantic segmentation processes all the patch embeddings of a singe image and tries to classify each of them, i.e., both the transformation and the classification are done at the patch-level. Such consistency enables us to interpret it by the idea of PCA. In other words, as models are typically only allowed to make direct information exchange within pixels or patches of a singe image, image segmentation is the most suitable task for applying PCA. Although based on semantic segmentation, our proposed interpretation framework can be applied to other dense prediction tasks. For example, we can allow that a class can be represented by more than one principal directions and each of these principal directions stand for an object or a segment, resulting in object detection and instance segmentation. However, it is challenging to adopt our interpretation from pixel-level tasks to image-level tasks. But, we believe that compression is an inevitable perspective for many tasks and our work serves as a first yet a solid step toward developing a comprehensive interpretation framework.

Response to Question #1. Please refer to Response to Weakness #1.

Response to Question #2. Thank you for point out that our previous proofs are difficult to comprehend. To address this issue, we now adopt a classical perspective of low-rank approximation to prove that class embeddings are transformed to be the principal directions by the attention operations. Please refer to the Interpretation Outline section of our Common Response for our rephrased thoughts and conclusions. In short, we prove that the principal directions are a good low-rank approximation to the patch embeddings in terms of coding rate. Therefore, we replace equation (20) with $\min_{\boldsymbol{Q}} |R(\boldsymbol{Z})-R(\boldsymbol{Q})|$ $\text{s.t}.$ $\text{rank}(\boldsymbol{Q})=C$, which is a more intuitive objective motivated by the idea of low-rank approximation. By the geometric interpretation of the coding rate, we will have that $R(\boldsymbol{Z}) \geq R(\boldsymbol{Q})$, constraining that each class embeddings lie in the subspace of patch embeddings. Therefore, the objective becomes $\max_{\boldsymbol{Q}} R(\boldsymbol{Q})$, and it derives a self-attention-like operation on $\boldsymbol{Q}$. For keeping the constraint and building up the connection between $\boldsymbol{Q}$ and $\boldsymbol{Z}$, we can replace the first and the second $\boldsymbol{Q}$ in the self-attention $\boldsymbol{Q}(\boldsymbol{Q}^T \boldsymbol{Q})$ with $\boldsymbol{Z}$ and get $\boldsymbol{Z}(\boldsymbol{Z}^T \boldsymbol{Q})$, a cross-attention-like operation. Notice that the output of such an operation always lies in the subspace of $\boldsymbol{Z}$. It is easy to prove that the scaled leading $C$ principal directions are an optimal solution for the above low-rank approximation objective. Although we believe that current line of thought is intuitive, we are willing to provide detailed proofs if reviewers request.

[1] Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, Yi Ma. NeurIPS 2020.

Dear Reviewer V3V2

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC

Thank you for your valuable suggestions. As the reviewer suggested, we have further validated our proposed interpretation and evaluated the effectiveness of our DEPICT on a set of new experiments on datasets Cityscapes and Pascal Context, including ablating the variants, testing the robustness under parameter perturbation and evaluating GFLOPs. We report the performance of our modifications to DEPICT, their justifications, detailed experimental results and an outline of our interpretation in our Common Response.

Here, we address the concerns listed by reviewer MHpG.

• First of all, we evaluate the GFLOPs, the amount of floating point operations, of only the decoders for fair comparison. Compared with Segmenter's decoder (Mask Transformer), the GFLOPs of our DEPICT is 2/3/4/10 versus Mask Transformer's 2/6/22/37 when both based on ViT-Tiny/Small/Base/Large. Compared to MaskFormer's decoder, the GFLOPs of our DEPICT is 10 based on ViT-Large; whereas it is at least 270 for MaskFormer based on Swin-Large (total: 375 GFLOPs [1] - Swin-L: more than 104 GFLOPs [2]). These evidences demonstrate that our DEPICT indeed serves an extremely light-weight decoder with comparable even better mIoU. Specifically, we must express our apologies for a vital typo that we mis-spelled "Mask Transformer" as "MaskFormer" in our submitted paper. Considering that, "Mask Transformer" has been widely used in various papers, we would use "Segmenter's decoder" instead in the final version of our paper.

• Then, we report our newly added experimental results on Cityscapes and Pascal Context. Due to limited time during rebuttal phase, we did not report the results of ViT-Large based DEPICT on these two datasets, but we will get it done soon (during the discussion period or at least in the final version). Based on ViT-Small, we can observe that our DEPICT outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context.

• Third, as suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1% mIoU with at least 270 GFLOPs; whereas our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons. In addition, we now report better performance of our newly modified DEPICT, compared to Segmenter (the only fair comparison). Details can be found in our Common Response.

• For applying our interpretation to more black-box methods, it seems a potential but challenging future work. The most thorny part is prevalent methods typically involving a pixel decoder (see Fig.3 of [3]), resulting in extremely more GFLOPs and parameters, and remaining uninterpretable from our proposed perspective. Our work implies that there must exist more efficient methods without the burdensome pixel decoder.

Most importantly, as suggested by the reviewer, we add several interesting experiments to confirm that our DEPICT indeed performs as we claimed. Here, we would like to list all current evidences.

• We strictly implement the MSSA block (see Line 171 for previous implementation) and remove the ISTA block (see Line 197 for context), enabling us to conclude that all DEPICT does is maximizing the projected coding rate on subspaces spanned by leading principal directions or minimizing it on subspaces spanned by remaining principal directions. By the sign of the learned step-size parameter, it is a white box for us to find out whether a specific block is maximizing it or minimizing it.

• According to above point, all the learned step size should be positive if we let all blocks model the ambient space rather than low-dimensional subspaces. We conduct a set of experiments in this extreme case by setting $\text{heads}=1$ and $\text{dim}$_$\text{head}=\text{dim}$, and the result is as expected.

• We visualize all the class embeddings output by a cross-attention block across images and find that principal directions lies in a union of $C$ orthogonal subspaces.

• During the inference phase, we generate the $\text{heads}$ random orthogonal matrices per image and use them to transform the parameter matrices of each head individually. We observe that DEPICT based on various ViT backbones shows no accuracy drop under such a perturbation. Furthermore, we generate one random orthogonal matrix to transform the parameter matrix of the entire block. Despite parameters are perturbed across heads, DEPICT based on various ViT backbones still shows limited drop in mIoU (less than 3% for ViT-L based DEPICT). Such robustness strongly validate the interpretation that our attention blocks essentially model subspaces thus performing orthogonal transformations on their learned bases will not lead to collapsed performance.

We hope the clarifications above could resolve your concerns on our work. Please let us know if any further clarification is needed.

[1] Per-Pixel Classification is Not All You Need for Semantic Segmentation. Bowen Cheng, Alex Schwing, Alexander Kirillov. NeurIPS 2021.

[2] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. ICCV 2021.

[3] Transformer-Based Visual Segmentation: A Survey. Xiangtai Li, Henghui Ding, HaoboYuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, Chen Change Loy. TPAMI 2024.

[4] White-box transformers via sparse rate reduction. Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, Yi Ma. NeurIP 2023.

Dear Reviewer MHpG

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC

Thank you for your valuable comments. It is indeed that the meta-architecture we investigated sounds not representative enough for a vast range of Transformer-based decoders. However, the insights of our work is capable to be generalized to partly interpret them.

First of all, we express our sincere appreciation for pointing out our mistakes. We will adopt the meta architecture proposed by [1] (see its Fig.3) as the truly representative one and admit that our interpretation is investigating a simplified one in the paper. Additionally, we will replace "MaskFormer" with "Segmenter" in the table. The decoder of Segmenter is called "Mask Transformer" but we mistakenly spelled it. Considering that, "Mask Transformer" has been widely used in various paper, we would not to use it in the final version of the paper in order to avoid naming ambiguity.

Now, we focus on reporting how we attempt to address the concerns listed by reviewer.

• We would like to show that addressing a limitation of our current interpretation will result in replacing class embeddings with segment/instance embeddings. That is, using class embeddings couples each class with a principal direction, such that each class strictly corresponds to one principal direction. However, an image typically contains a small part of all classes and each class allows rich intra-class variance thus demands more than one principal directions to representing itself. Therefore, it is more reasonable to introduce segment or instance as a finer-grained and more flexible concept to replace the current role of class. As evidence, we find that there are still higher correlations among class embeddings output by our DEPICT. Although such correlations can be significantly eased by using cross-attention, it leads to a drop in mIoU. Additionally, Mask2Former reports that making query features learnable raises mIoU by 1.8% (see its Table 4. (b)). From the perspective of our work, it is because that the attention block performs a gradient descent step and a good initialization to start would be beneficial.

• As suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we have to argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1% mIoU with at least 270 GFLOPs; whereas our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons. As for the linear decoder, it is not scalable not only on layer backbone but also to trade the amount of parameters for better mIoU. For example, it lags behind our DEPICT by 1.8% mIoU when based on ViT-Large and simply adding more linear layers does not change the performance. (These results are based on our newly modified DEPICT, and have reported in our Common Response.)

• We actually find that adding constraint $\text{heads}*\text{dim}$_$\text{head}=C$ is unreasonable, as its derivation considered only one image whereas there are various principal directions across images of the whole dataset. Therefore, we have removed this constraint. Then, we conduct experiments to evaluate the impact of using a ranging number of heads, and find that using a relatively small number of heads (i.e., it is 3 for ADE20K) leads to the best performance, implying that a tighter lower bound leads to less effective training.

• We report more experimental results on Cityscapes and Pascal Context. Limited by time, We did not report the result of ViT-Large based DEPICT on these two datasets, but we will get it done soon (during the discussion phase or at least in the final version). Our DEPICT based on ViT-Small can outperform Segmenter by 7% mIoU on both Cityscapes and Pascal Context.

We hope that our clarification points above could resolve your concerns on our work. Please let us know if any further clarification is needed.

[1] Transformer-Based Visual Segmentation: A Survey. Xiangtai Li, Henghui Ding, HaoboYuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, Chen Change Loy. TPAMI 2024.

Dear Reviewer ZoMh

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC

Thank you for your appreciation of our work. As suggested, we add experiments on more datasets, including Cityscapes and Pascal Context, and find that our DEPICT performs consistently better that of its black-box counterparts, i.e., the decoder of Segmenter. And we would like to compare our DEPICT to more advanced methods, showing that our DEPICT serves as an extremely light-weight decoder with acceptable lower mIoU. Specifically, we express our apologies for a typo that mis-spelled "Mask Transformer" as "MaskFormer" in the paper. Since that "Mask Transformer" has been widely used in various papers, we would use "Segmenter's decoder" instead in the final version of our paper.

As reported in details in our Common Responses, we have provided evaluations on more datasets to improve the empirical evaluations as Reviewer CNaF suggested. To make a fair comparison, we strictly implemented the MSSA block (see Line 171 for previous implementation) and removed the ISTA block (see Line 197 for context). These experimental results show that our DEPICT now consistently outperforms Segmenter's decoder with further reduced amount of parameters. While the white-box nature of our DEPICT has been strengthened by the above-mentioned modifications, we further conduct more experiments to validate our interpretation.

Here, we focus on reporting how we have addressed the concerns listed in the review comments.

As raised by reviewer, the idea of quantifying compression by coding rate is indeed not the novelty of our work and, we did not claim that as one of our novel contributions. To make our novel contributions more clear, we would like to make the following interpretation (or clarification). Compared to [1], our novelty and distinct contributions are:

• We introduce the idea of compression to design an interpretable semantic segmentation and propose an interpretation framework from the perspective of the classical PCA.
• For the first time, we derive the cross-attention variant of the MSSA block and we conduct experiments to evaluate the performance of its strict implementation, showing improved performance (also see B.1.1 of [1] for its relaxed implementation).

Second, we would like to report more experimental results on datasets Cityscapes and Pascal Context. Due to limited time during rebuttal phase, we do not report the results of ViT-Large based DEPICT on these two datasets at this moment, but we will get it done soon (during the discussion, or at least in the final version). We report the results based on ViT-Small, and show that our DEPICT again outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context.

Third, as suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we have to argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1% mIoU with at least 270 GFLOPs; while our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons.

• For Question #1: "Is it possible to adapt PCA to CNN-based models?" This is an insightful but challenging question. It seems to us that the answer is yes. According to [2], a self-attention layer can express any convolutional layer. Conversely, a self-attention operation may be decomposed into convolutional filters. Additionally, in fully convolutional networks, classification presents as features output by the last layer. In other words, there seems no difference between extracting features and giving classification, for that is right the reason allowing us to introduce PCA for interpretation.

• For Question #2: we actually find that adding constraint $\text{heads}*\text{dim}$_$\text{head}=C$ is unreasonable, as its derivation considered only one image whereas there are various principal directions across images of the whole dataset. Therefore, we removed this constraint. Then, we conduct experiments to evaluate the impact of using a ranging number of heads, and find that a relatively small number of heads (i.e., 3 for ADE20K) renders the best performance, implying that a tighter lower bound leads to less effective training. Using a large number of heads not only causes a heavier occupation on GPU memory but also results in inferior performance. Please let us know if we misunderstand your question. Thanks so much!

We hope that our point-by-point responses above have resolved your concerns on our work and, highly appreciate your encouragement in the comments and the pre-rating. Please let us know if any further clarification is needed.

[1] White-box transformers via sparse rate reduction. Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, Yi Ma. NeurIP 2023.

[2] On the relationship between self-attention and convolutional layers. Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi. ICLR 2020.

Dear Reviewer CNaF

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC

We highly appreciate the reviewers' acknowledgment on our rebuttal as well as the efforts of AC. We are also appreciate the feedbacks from all reviewers, which help us to the quality of our work. In particular, we appreciate the suggestions regarding the experiments. Since the beginning of the rebuttal and discussion period, we started to conduct additional experiments and now all the aforementioned experiments are completed. To be specific:

• We measured the coding rate of subspaces modeled by the decoder across layers and found that the coding rate gradually concentrates on these modeled subspaces, validating that the decoder operates as our interpretation suggests.

• We tried to segment the image using the exact principal directions rather than the class embeddings calculated by the decoder. Surprisingly, we found that the principal directions serve as excellent classifiers, as their segmentation clearly highlights objects that the trained models failed to detect.

• We completed the required experiments on the Cityscapes and Pascal Context datasets, and are quite pleased to report that DEPICT remains strongly comparable to, and in some cases even surpasses, Segmenter on these two datasets.

In the final version of the paper, we will include all the experiments reported during the rebuttal and discussion period.

Below, we present the comparative results of DEPICT and Segmenter across three datasets. Note that although we did not have time to fine-tune the hyperparameters of DEPICT on the latter two datasets, DEPICT remains strongly comparable to, and even exceeds, Segmenter. In particular, on Pascal Context, DEPICT based on ViT-B outperforms Segmenter based on DeiT-B.