MaXTron: Mask Transformer with Trajectory Attention for Video Panoptic Segmentation

W2: Some components are modified with slight changes.

We thank the reviewer for the question. We respectfully disagree with the reviewer and clarify the concerns of each component as below:

W2.1: Axial-Trajectory Attention

We respectfully disagree with the reviewer with the point that 'the Axial-Trajectory Attention manipulates the set of tokens fed into transformer attentions, which has been widely used in lots of different areas'. As far as we are concerned, all attention mechanisms manipulate the set of tokens thus it is unfair to blame the design on this. Most importantly, it is non-trivial to adapt trajectory attention to video segmentation task. Though different self-attention mechanisms have been explored a lot in video classification, few works try to explore it in video segmentation due to the intolerant complexity. Our novel design effectively addresses this key challenge and improves within-clip consistency. Please refer to C1 in common concerns for more discussion on this. Finally, we kindly request the reviewer to re-evaluate our contributions. If the reviewer insists on the point that it is trivial to design axial-trajectory attention, we kindly request the reviewer for any reference papers that have ever successfully done so in literature.

W2.2: Cross-clip Tracking Module (Comparison with VITA)

Our cross-clip tracking module is not a simple modification from VITA. We clarify the detailed differences compared to VITA below:

1. VITA introduces an additional set of video object queries on top of the existing frame object queries, while MaXTron instead directly manipulates the clip object queries.

2. VITA contains both an encoder and a decoder to encode the information from frame object queries and decode the information from them via video object queries. As a comparison, MaXTron only employs a simple encoder to aggregate information from clip object queries.

Concretely, the operation flow of VITA is: frame query self-attention $\rightarrow$ frame query feed-forward network $\rightarrow$ video query cross-attention to frame query $\rightarrow$ video query self-attention $\rightarrow$ video query feed-forward network

The operation flow of cross-clip tracking module in MaXTron is: clip query trajectory-attention $\rightarrow$ clip query Temporal-ASPP

As can be seen, the proposed cross-clip tracking module enjoys a much simpler pipeline compared to VITA.

3. VITA uses their proposed video object queries to predict both the whole video mask and class directly, while MaXTron exploits clip object queries to predict clip mask separately (the final whole video mask is obtained by concatenating the clip masks) and predicts the class by weighted mean of aligned clip queries, motivated by the fact that the aligned clip queries should have consistent class prediction. We argue that the proposed per-clip mask prediction scheme leads to better video mask prediction, since it is challenging to use only one set of video queries to directly predict the mask for the whole video. We provide additional ablation study on this to support our claim by experimenting with MaXTron w/ Video-kMaX with ResNet-50 as backbone on VIPSeg:

4. Finally, VITA introduces additional similarity loss after introducing their video object queries to better supervise the training, while MaXTron does not need that. Besides, VITA takes frame object queries from the last three layers of the transformer decoder as input, while MaXTron only needs the clip object queries from the last layer.

In the end, we provide experimental results when replacing our cross-clip tracking modules with VITA below (all based on the same Video-kMaX baseline to ensure a fair comparison):

As shown in the table, our cross-clip tracking module is faster in training, more lightweight in params and FLOPs and achieves better performance compared to VITA.

Q1: FLOPs and FPS of MaXTron.

We thank the reviewer for the question. Please refer to common concerns C3 for the FLOPs and FPS comparison.

Q2: Statistical significance.

We thank the reviewer for the question. Yes, all experiments are conducted 3 times and the reported number is the mean of the results. Please note that we do not report this statistical significance as most works in this literature do not report this in their papers.

We thank Reviewer tzju for the review, and we address the concerns below.

W1: Some components are borrowed from literature.

We thank the reviewer for the question. We respectfully disagree with the reviewer and clarify the concerns of each component as below:

W1.1: Multi-scale Deformable Attention

Multi-scale Deformable Attention (MSDeformAttn) have been introduced in [1], and used in most recent works. We introduce MSDeformAttn in our within-clip tracking module for a fair comparison to other state-of-the-art methods. Most importantly, we show below that though MSDeformAttn brings improvement to performance, the most significant improvement is still brought by our own proposed axial-trajectory attention.

Firstly, we provide additional results on VIPSeg with ResNet-50 as backbone and ablate on the effectiveness of multi-scale deformable attention and the proposed axial-trajectory attention in terms of VPQ:

As can be seen, when compared with the baseline Video-kMaX, adding MSDeformAttn only brings 1.8 VPQ improvement, adding axial-trajectory attention brings 2.2 VPQ improvement. When combining together, we get the final 3.4 VPQ improvement. We also emphasize that Video-kMaX with axial-trajectory attention alone already achieves significant improvement compared to other state-of-the-art methods (Video-kMaX with axial-trajectory attention alone 44.9 vs. current state-of-the-art DVIS 39.2). Please refer to Tab. 5 for more comparisons with different within-clip tracking module designs.

Secondly, for video instance segmentation, please note that Tube-Link already contains MSDeformAttn in their own clip-level segmenter design and we simply add the proposed axial-trajectory attention after their MSDeformAttn. As a result, all performance gains on the near-online setting are brought by the proposed axial-trajectory attention. To be concrete, it brings 0.6 AP, 4.4 $AP^\text{long}$, 5.8 AP improvement when evaluating on Youtube-VIS-21, Youtube-VIS-22 and OVIS dataset with ResNet-50, ResNet-50, and Swin-L as backbone, respectively. Please refer to Tab. 2, 8, 9 and 10 for more comparisons.

In summary, MSDeformAttn does help with the within-clip tracking module, but it is not the core component in it and we do not claim it as our novelty as well. Most importantly, even without MSDeformAttn, our model still brings non-trivial improvement.

W1.2: Transformer Decoder

We kindly request the reviewer for clarification on the point that 'Transformer Decoder is borrowed from the literature'. As far as we are concerned, the transformer decoder is an essential and non-removable component for modern video segmentation models and all other works use it. If the reviewer insists on this point, we kindly request the reviewer for more clarifications, e.g., why the follow-up works should not (and never) use the Transformer Decoder.

W1.3: ASPP

ASPP is introduced by [2], and originally operates on the dense pixels in the spatial domain. As far as we are concerned, we are the first to extend it to the temporal domain. Motivated by its property in capturing local information, we extend it to operate on the clip object query to reason about the temporal information. This is a non-trivial adaptation of ASPP.

Besides, Tab. 6 (6) in the paper shows that even without ASPP, our usage of trajectory attention on clip object query still brings performance improvement due to the capture of global cross-clip interactions.

[1] Zhu, Xizhou, et al. "Deformable DETR: Deformable Transformers for End-to-End Object Detection". ICLR 2021 (Oral)

[2] Chen, Liang-Chieh, et al. "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs". TPAMI 2017

I have read the authors’ response and the reviews from fellow reviewers.

• The main concern that has not fully been addressed is the significance of the claimed contributions. For the past few years, numerous works have been published to address the quadratic computation issue of self-attention. There are many approaches that can handle the CUDA OOM issue such as reducing the scope of attention, decomposing the attention, reducing the number of tokens by taking hierarchical approach, and so on. A lot of those works have already presented such approaches, and they also provide customized CUDA codes that actually makes the model feasible. Compared to those works, I believe the axial-trajectory attention of this paper is much of a naive extension of self-attention: limiting the number of visiting tokens. In order to prove its effectiveness, the authors should have provided thorough analysis of the module and a comparison between temporal extension of existing transformer variants, not only the MSDeformAttn which does only 2D spatial encoding. To list a few, here are some of the references that I believe that could have been simply extended to a spatio-temporal version, and be applied to the VPS task.
  • Bertasius et al. Is Space-Time Attention All You Need for Video Understanding
  • Ramachandran et al. Stand-Alone Self-Attention in Vision Models
  • Hassani et al. Neighborhood Attention Transformer
  • Beltagy et al. Longformer: The Long-Document Transformer
  • Xie et al. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
  • Pan et al. Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

Especially, since the first reference can be directly applied, what is the benefit (accuracy & efficiency?) of the proposed module over Bertasius et al?

• Thank you for pointing out that the authors do not claim ASPP as their novelty. However, I do not view the application of ASPP as an enough contribution. It is true that there are not many applications to the dense video pixel-level prediction tasks. However, given that the VPS task is not a significantly popular task, and ASPP is an extremely well-known module in the vision domain, I cannot agree with the authors that it can be considered as a major contribution. Indeed, it is very obvious that ASPP can be seamlessly integrated into any module. I do not mean that this work should not have used the transformer decoder. To clarify, I believe the contributions for the overall architectural design is limited, e.g., using the transformer decoder is to be expected.

• Thanks for providing the comparison to VITA. Despite I understand that the time can be limited for experimenting during the rebuttal period, my remaining concern is that it is only experimented on top of Video-kMaX.

• From the 15 GFlops reduction over VITA, the authors mentioned that the presented module is more computationally efficient. Then, as shown in Table C3, it seems like MaXTron is extremely heavier than Video-kMaX (more than 30% increase).
  • What’s the GFlops and FPS of other state-of-the-art methods such as TarVIS, VITA, DVIS? Referring to the DVIS paper, DVIS has much less params even than Video-kMaX (which is much lighter than MaXTron).
    • I believe the FPS comparison can be reported if they are experimented on VIS benchmarks.
    • Since MaXTron also provides accuracies on VIS datasets, it might be easier and more straight-forward to compare on the VIS benchmarks.

We thank Reviewer tzju for the follow-up reviews, and we address the concerns below.

T1 The main concern that has not fully been addressed is the significance of the claimed contributions.

We thank the reviewer for the question and the provided references. We are happy to cite and add a brief discussion about them in the revised version. Before we start, we would like to clarify that the main aim of this work is not just designing an efficient attention mechanism. Our core objective lies in enhancing the tracking ability of clip-level segmenters, encompassing improvements on both within-clip and cross-clip levels. The introduction of axial-trajectory attention is a non-trivial exploration in advancing this specific pursuit.

T1.1 Should compare not only the MSDeformAttn which does only 2D spatial encoding

We note that in our previous rebuttal, we offered the detailed comparison to MSDeformAttn mainly to address your previous concern "MSDeformAttn is borrowed from literature" and showed that the proposed axial-trajectory attention brings more significant improvement than MSDeformAttn. We are not merely comparing with MSDeformAttn, but instead we have carefully compared with the mostly related works to support our claim (enhancing the tracking ability of clip-level segmenters via axial-trajectory attention) as shown below:

• Video segmentation:
  • TarVIS (for within-clip tracking comparison), VITA (for cross-clip tracking comparison)
• Video classification
  • Divided Space-Time Attention [1].

[1] Bertasius et al. Is Space-Time Attention All You Need for Video Understanding

We thank Reviewer LGcv for the review, and we address the concerns below.

W1: Writing should be improved. confusing about the implementation details.

We thank the reviewer for pointing this out and for the suggestions. Please refer to Q3, W1 and W2 in response to Reviewer TBSQ in case there are any confusions regarding axial-trajectory attention, trajectory attention and Temporal-ASPP, respectively.

Additionally, as promised in our abstract, we will open-source all our implementations and checkpoints, allowing the community to scrutinize the details (for both training and inference).

W2: Provide the cost, e.g. training cost and inference speed

We thank the reviewer for pointing this out. Please refer to common concerns C3 for detailed training cost and inference speed.

W3: Deeper analysis of how the association capability is improved by the proposed modules.

We thank the reviewer for pointing this out. The association capability is improved with two modules: within-clip tracking and cross-clip tracking.

In the within-clip tracking module, MaXTron effectively exploits axial-trajectory attention to reason about the pixel-level temporal association. Concretely, axial-trajectory attention finds the probabilistic path of each pixel along time as shown in Fig. 1. It then aggregates information along this probabilistic path. Intuitively, this helps to capture pixel correspondence and avoids inconsistent prediction within a clip. As a result, the within-clip tracking module brings 3.4 and 3.5 VPQ improvement on VIPSeg, with ResNet50 and ConvNeXt-L, respectively (see Tab. 1 in the paper).

In the cross-clip tracking module, MaXTron exploits trajectory attention along with Temporal-ASPP to capture both the global and local cross-clip connections. Intuitively, the per-clip prediction is not perfect and simply associating them with Hungarian Matching as done in most existing online/near-online methods might lead to incorrect matching. This might further lead to problems such as ID switching or miss detection. By introducing the cross-clip tracking module, we aim at aligning the clip object queries more precisely by modeling the connections between them. As a result, our offline method with the proposed cross-clip tracking module is able to provide more consistent prediction at video-level and thus brings an additional 0.6 and 0.9 VPQ improvement on VIPSeg, with ResNet50 and ConvNeXt-L, respectively (see Tab. 1 in the paper).

We thank Reviewer tzTu for the review, and we address the concerns below.

Q1: What if we change the number of frames within one video clip?

We thank the reviewer for bringing up this question. We observe performance change when varying the number of frames within one video clip. Most likely, it is due to the fact that the off-the-shelf clip-level segmenter itself performs differently with different clip sizes. Concretely, for Video-kMaX and MaXTron w/ Video-kMaX on VIPSeg with ResNet50 as backbone in terms of VPQ, we have:

As we can see, with the increase of clip size, Video-kMaX performance gradually decreases. However, both our MaXTron online (w/ within-clip tracking module) and MaXTron offline (w/ within-clip + cross-clip tracking module) models bring steady and consistent improvements. We hypothesize that the performance difference of Video-kMaX is due to the lack of a temporal module in its framework and the temporal connections are forced to be modeled by the transformer decoder only. As a result, with the increase of clip size, the transformer decoder can not handle the features properly, leading to the decrease in performance. Our axial-trajectory attention alleviates this problem and achieves slightly larger improvement when clip size = 2. Besides, our cross-clip tracking module is less sensitive to clip size as it directly learns to model video-level input.

Q2: Does the proposed solution perform differently if we (1) process all continuous frames or (2) only process key frames (with some down-sampling temporally)?

We thank the reviewer for bringing up this question. We would like to clarify the challenges of dense prediction is to label all pixels in the video, and it is very challenging to only process key frames and propagate the key frame results to other frames during testing.

However, as suggested in C2 in common concerns where we discuss possible failure cases, one main challenge that still remains for MaXTron is to model fast-moving objects. As a result, we hypothesize that MaXTron will perform worse if the video itself contains fewer frames, which might be caused by down-sampling with a large factor in the temporal domain. This is partially supported by our experiments where we train the model with discontinuous sampled frames (specified by the frame sampling range below):

Above experiments are done with MaXTron w/ Video-kMaX on VIPSeg with ResNet-50 as backbone, clip size = 2 and are evaluated in terms of VPQ. As shown in the table, training with continuous frames brings the best performance. While slightly increasing the sampling range does not affect the performance a lot, increasing it to $\pm10$ greatly hampers the learning of the within-clip tracking module.

W1: Report some failure cases.

We thank the reviewer for pointing this out. Please refer to common concerns C2 for the discussions on common error patterns of MaXTron.

We thank Reviewer TBSQ for the review, and we address the concerns below.

Q1: Main contribution?

As properly phrased in the submission, we never claim trajectory-attention as our main contributions. Instead, as mentioned in the common concerns C1, our contribution lies in the fact that we are the first to successfully adapt trajectory-attention to the dense pixel prediction tasks, by designing axial-trajectory-attention for tracking pixel movement in H -axis and W-axis separately to model the within-clip temporal cues as well as exploiting trajectory-attention on clip object queries to model the cross-clip temporal connections.

To the best of our knowledge, none of the existing works have successfully explored trajectory-attention for dense prediction tasks and very few works ever explored temporal attention for dense video prediction. As shown in the experimental results, our attempt has successfully led to a significant boost. We kindly request the reviewer to re-evaluate our contributions. If the reviewer insists on the point that it is limited and insufficient to design trajectory-based within-clip and cross-clip modules for dense prediction, we kindly request the reviewer to elaborate more on the reasons.

Q2: Examples of attention maps and how to get the trajectories or the trajectories might not be perfect.

Q2.1 More examples of attention maps

Please see the revised version for more examples of attention maps of selecting either dynamic or static points as reference points. Concretely, please refer to Fig. 9, 10, 11, 12 and Sec. F. 1 in Appendix for the detailed discussion.

Q2.2 How to get the trajectories?

Please see Q3 for detailed explanation.

Q2.3 Trajectories might not be perfect.

Please see the revised version for failure cases (Fig. 13 and 14). Besides, as suggested by the discussions in C2 in common concerns, the trajectories might also fail when the selected reference points are not discriminative enough.

Q3: How to get the probabilistic path of a point between frames?

Overview: We first provide an overview of how to get the probabilistic path. Conceptually, the affinity logits of pixels across different frames are computed within the process of axial-trajectory attention. As a result, simply exploiting the computed attention map can bring us the visualizations of the probabilistic path. For your convenience, we now explain it in detail below.

We take H-axis trajectory attention for example. We first reshape the feature map extracted by the backbone $F \in \mathbb{R}^{\mathit{T} \times \mathit{D} \times \mathit{H} \times \mathit{W}}$ into $F_{h} \in \mathbb{R}^{\mathit{W} \times \mathit{TH} \times \mathit{D}}$ before conducting H-axis trajectory attention. In this way, we obtain a sequence of $\mathit{TH}$ pixel features $x_{th} \in \mathbb{R}^{D}$.

Then the H-axis trajectory attention can be divided into two steps.

1. Trajectory along H-axis (Eq. (2) in the paper): Given a reference point at a specific time-height th position, we compute its possible time-height locations at each frame separately followed by extracting per-frame information (i.e., trajectories) with the computed probability.

2. Temporal attention along the trajectory (Eq. (4) in the paper): Given the trajectories, we aggregate the information along them using 1D time attention to reason about temporal connections.

The construction of the probabilistic path lies in step 1. Concretely, we use linear projection to project all features $x_{th}$ into their corresponding queries, keys and values. Then for a given new time $t'$ (i.e., the frame index), the trajectory tokens are computed by (Eq. (2) in the paper): $\widetilde y_{tt'h} = \sum_{h'}v_{t'h'} \cdot \frac{\exp{\langle q_{th},k_{t'h'} \rangle}}{\sum_{\overline{h}}\exp{\langle q_{th}, k_{t'\overline{h}}\rangle}}$.

Intuitively, the operation in the numerator $\langle q_{th},k_{t'h'} \rangle$ yields an attention map $A \in \mathbb{R}^{\mathit{T} \times \mathit{H}}$, which computes the similarity logits of the reference point th and pixel features at all other possible TH locations $t'h'$.

As a result, to draw the H-axis trajectory attention map across frames, we first pick a reference point th (e.g. the basketball in Fig. 1) followed by computing the attention map $A$ as above. We can then set the new time point to $t+1$, $t+2$, ... $t+k$ and retrieve the corresponding position at H-axis iteratively. By putting them together, we get the trajectory path along H-axis. Similarly, the same operation is conducted for the W-axis to obtain the trajectory path along the W- axis. As explained in the caption of Fig. 1, for the purpose of visualization, we multiply the H-axis and W-axis trajectory attentions to visualize the trajectory of the reference point over time (i.e., a bright point corresponds to a high attention value in both the H- and W-axis trajectory attention).

C3: From reviewers LGcv, tzju, Training cost and inference speed

We thank the reviewers for the questions. Since most prior works in this literature do not report these numbers, we can only provide the comparison between MaXTron and our baseline off-the-shelf clip-level segmenters here in terms of training GPU days, model parameters, GFlops and FPS.

We first compare the effect of deploying the proposed within-clip tracking module to the baseline Video-kMaX as follows:

As can be observed from the table, the proposed within-clip tracking module contains very few parameters (marginal 9.1M). It brings reasonable increase in training time and GFlops, since it is compute-intensive to conduct attention-based operation on the high-resolution dense pixel feature maps. We note that MSDeformAttn and Axial-Trajectory Attn brings roughly the same amount of computational cost.

Now, we carefully compare the effect of deploying the proposed cross-clip tracking module to Video-kMaX. Specifically, we compare MaXTron w/ Video-kMaX (denoted as Ours) vs. VITA w/ Video-kMaX (denoted as VITA) in the following table:

As shown in the table, our cross-clip tracking module is more lightweight compared to VITA and achieves better performance, which is brought by the much simpler design of our cross-clip tracking module. We refer to Q.2.2 to Reviewer tzju for the detailed comparison. FPS is not reported here since offline inference is done on CPU (following the literature) due to the high resolution ($769 \times 1345$) and long video length (24 frames in the table).