GLOMA: Global Video Text Spotting with Morphological Association

Why not simply follow MOT?

As SOT is not suitable for VTS, we here only discuss MOT.

The differences between MOT and VTS:

This paper focuses on the notion that while following the established pipeline in MOT has proven useful in previous studies on VTS, there are unique aspects of video text scenes that previous research has overlooked. These differences, compared to objects in MOT or SOT (such as pedestrians, vehicles, or general objects), are essential for enhancing VTS performance. As discussed in lines 037-046, two key distinctions arise: (a) The visual appearance of texts remains more stable in VTS. (b) While the shapes of texts are more stable, their motion can be quite dramatic. Texts do not experience limb deformation; instead, most deformations are due to perspective changes, with only a few instances of occlusion. As illustrated in Figure 4, a significant portion of texts moves so quickly that some have no overlapping area even between two consecutive frames.

Mainstream MOT Pipelines and the Introduction of Global and Morphological Information in VTS

A typical MOT algorithm considers both visual and positional information, or sometimes focuses solely on positional information (e.g., OC-SORT, ByteTrack). These algorithms often rely on features from the most recent frame, occasionally incorporating an updating scheme to maintain implicit attention to earlier frames. This approach is understandable, as visual features in MOT scenarios can be unstable over longer periods. For instance, a pedestrian may turn around between frames. Thus, it makes sense to prioritize the most recent features in MOT. In contrast, scene texts themselves do not change; any deformation is typically caused by camera motion. Introducing global information can effectively address issues such as motion blur in recent frames. Furthermore, in MOT, positional associations are primarily represented as IoU scores. This limitation arises because MOT datasets use bounding boxes defined by only two points (e.g., tlbr), which lack morphological information. Additionally, the frequent changes in limb position and pose further diminish the contribution of morphological details to the tracking process. This situation contrasts sharply with VTS. The shapes of scene texts do not change unless they are rarely occluded. This stability is why we can effectively utilize morphological information in VTS.

Better presentation about figures.

As indicated in the legends of Figure 1 and Figure 2, the colorful circles represent the embeddings used for association, which are extracted using RotatedRoIAlign based on the detection results (see lines 155-158). The detection results are represented as 4-point coordinate bounding boxes (see line 157). We perform the association between texts in Frame t and those in the global pool (see lines 158-161) to build trajectories. We will include this information in the figure captions for better clarification.

New insights

As discussed above, this paper introduces global matching for VTS while considering morphological information, given the unique features of VTS compared to MOT. We aim to inspire future research in this field to not only adopt new methods from MOT but also to focus more on the distinct characteristics of VTS. Furthermore, to our knowledge, there are no existing methods that use Wasserstein distance as a matching score. When we refer to "global," we do not mean the entirety of a video; instead, it pertains to the lifespan of a text instance. Our focus is not on comprehensively understanding the entire video but specifically on VTS. It is unnecessary to maintain tracking and embeddings for a text that is permanently out of view. As shown in Table 3, we conducted ablation studies on the sliding window size and found no performance gain when further increasing the window size, but it does increase computational cost. Theoretically, we could perform associations across a whole video, but the associated time cost would be prohibitive. Given the fact that previous studies in VTS do not focus much on designing morphological associations, we compare the Gaussion Wasserstein distance with L1/2 distance, the results are as follows:

Comparison with recent approaches

Since recent methods often use various additional training data (e.g., DSText, ArtVideo, BOVText, etc.), making direct comparisons can be unfair. In our paper, we will discuss these methods, include their results, and clearly indicate which ones use additional data for training.

Thanks for you reply!

I believe we should distinguish between the ideal definition of MOT and the existing methods within this field. Theoretically, VTS can be considered a sub-area of MOT, and I agree with that perspective. However, due to the limited distribution of current MOT datasets, existing methods have significant limitations and are not sufficiently generalized for direct application to VTS.

Most commonly used MOT datasets, such as MOT17, MOT20, TAO, and DanceTrack, primarily feature fixed-camera videos. Although some less mainstream datasets, like KITTI, include camera motion, the movements are generally not rapid. This differs significantly from the motion patterns of texts in VTS, where most movement is caused by camera motion, often resulting in irregular trajectories. In contrast, objects in existing MOT datasets tend to move themselves, albeit more slowly, and are affected by changes in pose.

The MotionTrack method you mentioned aligns with our earlier comment that "visual features in MOT scenarios can be unstable over longer periods." As a result, it seeks solutions based on long-range motion (something like a learnable Kalman Filter often applied in MOT), which tends to be more regular in existing MOT datasets, rather than continuing to rely on visual features.

Summary:

1. The challenges in VTS and MOT differ significantly:

Consequences:

• Motion Patterns: MOT heavily relies on positional association. Most MOT methods prioritize positional association scores over visual scores.
• Shape and Coordinate Patterns: MOT methods do not leverage morphological information.
• Occlusion and Visual Appearance: MOT often employs motion prediction techniques (e.g., Kalman Filter) to estimate movement during occlusions or relies on visual features from the last observed frame. In contrast, VTS can rely on global visual features or shape similarity. Additionally, the ablation study in Table 4 demonstrates that GLOMA relies more on visual features than on positional and morphological information. This contrasts with MOT methods, which typically depend more on motion cues.

2. VTT/VTS is only the name of this field. It can be thought of as multi-text tracking. However, this similarity in definition does not imply that current MOT methods are suitable for VTT/VTS. The key point is not whether VTT/VTS is a sub-field of MOT, but rather that current MOT methods do not adequately address the requirements of VTT/VTS.

Highlight temporal information and morphological features in VTS.

We agree with your suggestions and we will add some illustrations in Figure 1.

Gaussian Wasserstein distance

As illustrated in Figure 3, we present three cases where Gaussian Wasserstein distance outperforms the IoU score due to its consideration of morphological information. This method can be extended to other MOT scenarios if objects are represented as rotated boxes. However, currently mainstream MOT datasets do not take the angle of bounding boxes into account.

Equation 8

The normalization term helps mitigate the impact of the absolute values of the boxes' width and height. We can do a simple math derivation here. Assume that $\\theta_1$ is $0$ and $\\theta_2$ is $\\pi/2$, we can get:

1. Firstly:
   • \\sigma_1=\\begin{pmatrix}\\frac{w_1}{2}&0\\\\0&\\frac{h_1}{2}\\end{pmatrix}. Then, \\sigma_1^{1/2}=\\begin{pmatrix}\\sqrt{\\frac{w_1}{2}}&0\\\\0&\\sqrt{\\frac{h_1}{2}}\\end{pmatrix}.
2. Secondly:
   • \\sigma_2=\\begin{pmatrix}\\frac{h_2}{2}&0\\\\0&\\frac{w_2}{2}\\end{pmatrix}. And \\sigma_2^{1/2}=\\begin{pmatrix}\\sqrt{\\frac{h_2}{2}}&0\\\\0&\\sqrt{\\frac{w_2}{2}}\\end{pmatrix}.
3. Next:
   • \\sigma_1^{1/2}\\sigma_2\\sigma_1^{1/2}=\\begin{pmatrix}\\sqrt{\\frac{w_1}{2}}&0\\\\0&\\sqrt{\\frac{h_1}{2}}\\end{pmatrix}\\begin{pmatrix}\\frac{h_2}{2}&0\\\\0&\\frac{w_2}{2}\\end{pmatrix}\\begin{pmatrix}\\sqrt{\\frac{w_1}{2}}&0\\\\0&\\sqrt{\\frac{h_1}{2}}\\end{pmatrix}=\\begin{pmatrix}\\frac{w_1h_2}{4}&0\\\\0&\\frac{w_2h_1}{4}\\end{pmatrix}.
4. Then:
   • (\\sigma_1^{1/2}\\sigma_2\\sigma_1^{1/2})^{1/2}=\\begin{pmatrix}\\sqrt{\\frac{w_1h_2}{4}}&0\\\\0&\\sqrt{\\frac{w_2h_1}{4}}\\end{pmatrix}.
5. After that:
   • \\sigma_1+\\sigma_2 - 2(\\sigma_1^{1/2}\\sigma_2\\sigma_1^{1/2})^{1/2}=\\begin{pmatrix}\\frac{w_1}{2}+\\frac{h_2}{2}-2\\sqrt{\\frac{w_1h_2}{4}}&0\\\\0&\\frac{w_2}{2}+\\frac{h_1}{2}-2\\sqrt{\\frac{w_2h_1}{4}}\\end{pmatrix}=\\begin{pmatrix}(\\sqrt{\\frac{w_1}{2}}-\\sqrt{\\frac{h_2}{2}})^2&0\\\\0&(\\sqrt{\\frac{w_2}{2}}-\\sqrt{\\frac{h_1}{2}})^2\\end{pmatrix}.
6. Then:
   • $\\text{Tr}(\\sigma_1+\\sigma_2 - 2(\\sigma_1^{1/2}\\sigma_2\\sigma_1^{1/2})^{1/2})=(\\sqrt{\\frac{w_1}{2}}-\\sqrt{\\frac{h_2}{2}})^2+(\\sqrt{\\frac{w_2}{2}}-\\sqrt{\\frac{h_1}{2}})^2$.
7. According to $d^2=\\|\\mu_1 - \\mu_2\\|^2+\\text{Tr}(\\sigma_1+\\sigma_2 - 2(\\sigma_1^{1/2}\\sigma_2\\sigma_1^{1/2})^{1/2})$, assuming $\\mu_1=\\mu_2=(0,0)$, we have:
   • $d^2=(\\sqrt{\\frac{w_1}{2}}-\\sqrt{\\frac{h_2}{2}})^2+(\\sqrt{\\frac{w_2}{2}}-\\sqrt{\\frac{h_1}{2}})^2$. Expanding it, we get $d^2=\\frac{w_1}{2}- \\sqrt{w_1h_2}+\\frac{h_2}{2}+\\frac{w_2}{2}-\\sqrt{w_2h_1}+\\frac{h_1}{2}=\\frac{w_1 + w_2+h_1 + h_2}{2}-\\sqrt{w_1h_2}-\\sqrt{w_2h_1}$.
8. Now, for calculating $W$. We know that $W(b_1,b_2)=1-\\frac{\\alpha d}{f(\\sigma_1,\\sigma_2)}$, where $f(\\sigma_1,\\sigma_2)=(\\text{Tr}(\\sigma_1\\sigma_2))^{1/4}$.
   • First, find $\\text{Tr}(\\sigma_1\\sigma_2)$. Given \\sigma_1=\\begin{pmatrix}\\frac{w_1}{2}&0\\\\0&\\frac{h_1}{2}\\end{pmatrix} and \\sigma_2=\\begin{pmatrix}\\frac{h_2}{2}&0\\\\0&\\frac{w_2}{2}\\end{pmatrix}, we have \\sigma_1\\sigma_2=\\begin{pmatrix}\\frac{w_1h_2}{4}&0\\\\0&\\frac{w_2h_1}{4}\\end{pmatrix}. So, $\\text{Tr}(\\sigma_1\\sigma_2)=\\frac{w_1h_2 + w_2h_1}{4}$.
   • Then, $f(\\sigma_1,\\sigma_2)=(\\frac{w_1h_2 + w_2h_1}{4})^{1/4}$.
   • And $d=\\sqrt{\\frac{w_1 + w_2+h_1 + h_2}{2}-\\sqrt{w_1h_2}-\\sqrt{w_2h_1}}$.
   • Finally, substituting these into the formula for $W$, we get:
   • $W(b_1,b_2)=1-\\frac{\\alpha\\sqrt{\\frac{w_1 + w_2+h_1 + h_2}{2}-\\sqrt{w_1h_2}-\\sqrt{w_2h_1}}}{(\\frac{w_1h_2 + w_2h_1}{4})^{1/4}}$.
   • If we set $w_1=w_2=k*h_1=k*h_2$, we have:
   • $W(b_1,b_2)=1-\\frac{\\alpha\\vert\\sqrt{k} - 1\\vert}{\\left(\\frac{k}{2}\\right)^{1/4}}$

So that $W(b_1,b_2)$ is not influenced by the absolute values of w, h. α is expected to rescale the Wasserstein distance score into a suitable range. We set it as 1.0.

Embedding computation and updating mechanism

As discussed in lines 281-297, during inference, we maintain an embedding pool to store embeddings of text instances within a sliding window. For each new frame, we attempt to associate its text instances with those in the global pool. If a match is found, we update the track. If a new text instance doesn't match any in the pool, it indicates the start of a new trajectory. Table 3 presents ablation studies on the sliding window size. Since GLOMA is a tracking-by-detection process in VTS, it experiences minimal tracker drift issues.

Writing issues

We will add a transitional paragraph to clarify the necessity of GLOMA, emphasizing both global and morphological information. Additionally, we will refine Figure 1 to clearly illustrate more failure scenarios and enhance the clarity of the figure captions. Furthermore, we will include the recognition module in Figure 2, which was initially omitted for clarity.

The Use of Synthetic Data

At a high level, this paper aims to explore effective methodologies for video text spotting rather than merely striving for top performance, in other words, this is a method paper other than a data paper. Therefore, ensuring a fair comparison is crucial, as most methods in this field follow a similar pipeline: they pretrain on the COCO-Text dataset and then fine-tune on downstream datasets. While some studies utilize different datasets (e.g., BOVText, DSText, ArtVideo, etc.), our approach does not incorporate additional data in line with our main focus. We think ensuring fair comparison is more important, even though scaling the data often yields benefits.

The Choice of the Text Detector/Spotter

Furthermore, this paper aims to delve into end-to-end video text spotting, which necessitates consideration of performance across detection, tracking, and spotting tasks, rather than focusing solely on one or two aspects. We conducted early explorations using advanced architectures of text spotters, training from scratch while ensuring fairness by not using pretrained model weights. However, we found that the architectures of these text spotters were not well-suited for unified tasks. We appreciate your suggestion to include some scene text spotting methods in our related works section. We will make the necessary revisions to enhance the clarity and depth of our paper.

The proposed association method integrates global visual information, positional information, and morphological details, which are crucial in VTS scenarios. In contrast, TransDETR primarily adheres to the MOT pipeline, focusing on local matching and IoU-based positional association. Our approach places greater emphasis on the inherent characteristics of scene texts in videos.