LoTLIP: Improving Language-Image Pre-training for Long Text Understanding

Q1: Contributions are unclear.

Sorry for confusion. We reaffirm our core contribution:

• We are the first to explore how to improve understanding long texts in contrastive language-image pre-training, and also firstly design learnable text [CLS] tokens (corner tokens) for this purpose.

Meanwhile, we heartily believe that it is necessary to study how to design a text tencoder that is visually aligned and has the ability to understand long texts.

Q2: From an architectural perspective, there seems to be no major difference from [1, 2].

We are beg to differ with you on this matter. The core difference is not learnable tokens, but text [CLS] tokens and our designed attention mask mechanism. [1] and [2] both design learnable tokens in image encoder, which give few help to long-text understanding (refer to Reviewer NJUx, Q4). Moreover, directly introducing register tokens [1] to text encoder also does not improve the performance (-0.02 performance degradation), because register tokens is used to get rid of artifacts in images (This problem not exists in text). Our coner token can enhance its capability of long text understanding (+2.05 performance improvment)

As a concurrent work with our LotCLIP, we look forward to whether using [2] into text encoder can enhance CLIP's ability to understand long texts, but unfortunately its official code is currently inaccessible.

Q3: Is there any special reason or evidence that such a technique is especially beneficial for training with long captions?

This is really an interesting aspect. Coner token is a general technique but significantly enhances the understanding of long text. Moreover, our method shows a more significant performance improvement when applied to long captions (+2.05 performance improvment) compared to shorter ones (+0.18 performance improvment).

Q4: Corner tokens in different position

Thanks for you valuable suggestion. Corner tokens around [CLS] token may bring better long-text understanding capacity. After Text = 67.82%, Before [CLS] =68.49% (+0.67), After [CLS] = 68.63% (+0.81). In BERT architecture, the [CLS] token is always the first token of text input. Thus, around [CLS] token may have better performance.

Q5: Corner tokens in vision transformers

Thanks for your valuable suggestion. We implement the corner tokens in the image encoder and find that it provides less improvement (0.99% performance improvement) to long-text understanding compared to utilize corner tokens in the text encoder (2.01% performance improvement).

Q6: Statistics about the training data

As shown in the table 1 of global pdf, we report some statistics of our dataset and compare it with other long-text datasets.

Q7: Details on the MLLMs such as specific architecture and instruction information

We provide the hyper-parameter settings of the used MLLMs as shown in the table 2 of global pdf.

Q8: For the LAION and COYO datasets, how is a subset selected?

Following Stable Diffusion, we filter Laion and COYO to images with aesthetic_v2>5.0 and resolution ≥512 to about 70M preserved pairs.

Q9: Verification step for the extracted long captions.

Thanks for your suggestion. We verify the quality of long texts of three MLLM for IIW dataset, and compare with human annotated long texts. We utilize GPT4V for assessment the alignment between image and text, following Q-Bench [3] (IIW =4.51, InstructBLIP-Vicuna7B=2.80, LLaVA-v1.5-13B=3.45, ShareGPT4V-13B=3.48). They have good score and low hallucination. Training with long texts from multiple MLLMs may mitigate the inherent biases of one MLLM.

[3] Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

Q10: It is unclear whether the comparison is fair. More visual backbones with LiT training

We compared various methods as fairly as possible. Our long-caption dataset and designed corner token can consistently improve various artitectures. We use unsupervised ViT (DINO) and self-supervised ViT (MoCo-v3) as image encoder and counduct training with 3M scale dataset following LiT.

Q11: Overlap on training and evaluation dataset

There is no overlap in the images used for training and the ShartGPT4V evaluation, as all the images in our used ShartGPT4V evaluation are sourced from the SAM dataset.

Q12: Directly comparable to LongCLIP

Thanks for your valuable suggestion. Directly comparable to LongCLIP, our LotCLIP also has better performance. We implement LotCLIP to fine-tune pre-trained CLIP model using the 1M-scale dataset from ShareGPT4v, similar to LongCLIP.

Q13: Reason behind the names (Dora and LotCLIP)

We name the method as LotCLIP (Long Texts in Contrastive Language-Image Pre-training) and we name the dataset as Dora (Detailed Texts of Real Images). We will add the explainations in revision.

Q14: Explanation on the concept of Corner Tokens

Sorry for the confusion. We will add the supporting explanations of corner tokens in the updated version. Specifically, we add several learnable tokens from different initialization after [CLS] token, termed as corner tokens.

Q15: Training Details

We apologize for missing the training details, and we add these in table 3 of global pdf. We will fix this issue in the updated version.

I appreciate the authors' extensive efforts in addressing my concerns.

The rebuttal addresses many of the initial issues. In both the main paper and the rebuttal, the proposed framework is effective in tasks related to long texts.

---

However, my primary concerns on the technical novelty remain unresolved. Despite the improved empirical results, the architectural modification on the 'register tokens' seem marginal, particularly in the insertion of learnable tokens into a text encoder instead of a visual encoder. Could the authors further elaborate on this point?

Additionally, the modification of the attention mask in LotCLIP appears to offer marginal improvements for corner tokens in Table 3, diminishing its significance as a technical contribution. Moreover, the explanation for the improvements in long texts (Q3) is not sufficient for me. Could you share any insights behind these improvements?

In the training dataset and MLLMs, thank you for the further clarifications. I value the substantial volume of the generated data, both in terms of quantity and token lengths. However, I perceive the limited technical innovation, since the pipeline simply relies on standard MLLM inference for generating long texts.

I think that the focus of the technical contributions should primarily be on the modeling aspect, which currently seems underdeveloped in this version.

---

Further questions arising from the other reviewers' comments and the authors' responses.

(1) Are the corner tokens defined as separate 'text' elements rather than as learnable token parameters? According to the training process described in the authors' comment, corner tokens undergo the tokenization process alongside the original captions. Could you please clarify this?

(2) Are corner tokens still effective without the BERT's CLS token? For a more comprehensive analysis, I suggest (e.g., not mandatory) conducting an experiment, if feasible within the author-reviewer discussion period, that ablates the introduction of LotCLIP, replacing the BERT text encoder with an OpenAI-style text transformer. OpenAI counterpart disables the non-causal attention masks, and uses the last token as text pooling.

(3) In measuring long text understanding capabilities with images, the paper considers image-text retrieval. Are there any potential surrogate tasks for assessing understanding in conjunction with long texts and images?

Detailed experimental results

Below, we provide detailed experimental results in response to Q2, Q3, Q4, Q5, Q9, Q10, and Q12.

Q2: From an architectural perspective, there seems to be no major difference from [1, 2].

[1] Darcet et al., Vision Transformers Need Registers, in ICLR 2024.

Q3: Is there any special reason or evidence that such a technique is especially beneficial for training with long captions?

Q4: Corner tokens in different position

Q5: Corner tokens in vision transformers

We are really sorry that the results provided in the response of Q5 are not correct. We implement the corner tokens in the image encoder and find that it provides less improvement (0.58% performance improvement) to long-text understanding compared to utilize corner tokens in the text encoder (1.84% performance improvement).

Q9: Verification step for the extracted long captions.

We utilize GPT4V to assess the alignment between image and text, following Q-Bench [3]. The prompt we used is "{image}. Text: {text}. Please assist in analyzing whether the given text aligns with the given image. Please provide an integer score as a single number from 0 to 5 based on the alignment, without explanation.".

Q10: It is unclear whether the comparison is fair. More visual backbones with LiT training.

Q12: Directly comparable to LongCLIP

Looking forward to your feedback

We look forward to hearing from you, and we can further address unclear explanations and remaining concerns, if any.

Thanks for your feedback. We are encouraged by your appreciation on the effectiveness of LotCLIP in tasks related to long texts. And we are glad that our rebuttal addresses some of your concerns. Bellow, we address your remaining concerns and additional questions separately.

Q16: However, my primary concerns on the technical novelty remain unresolved. Despite the improved empirical results, the architectural modification on the 'register tokens' seem marginal, particularly in the insertion of learnable tokens into a text encoder instead of a visual encoder. Could the authors further elaborate on this point?

We are beg to differ with you on this matter.

a) Technical difference: Corner tokens and register tokens differ in more than just applying on different encoders. Although the corner tokens and register tokens are both learnable, they differ fundamentally in the following aspects:

• Aligning corner tokens to visual information: The features of corner tokens are utilized for contrastive learning to align with visual information, while register token outputs are not used for any purpose.
• Inserting corner tokens to text dictionary: The corner tokens are added into the vocabulary of tokenizer and converted to learnable embeddings, as widely used method in the NLP field, e.g., BERT.
• Making corner tokens diversity: Each corner token corresponds to identical token id in the vocabulary of tokenizer ensuring the disparity among corner tokens. Moreover, the interactions between corner tokens and other tokens are restricted by an attention mask mechanism. It promotes the corner tokens to learn diverse text informantion, which helps [CLS] token on image perception.

b) Motivation difference: The corner tokens are designed to assist [CLS] token in aggregating text token features, while the register tokens are designed to mitigate the high-norm outlier tokens in image patches(tokens). Besides, the high-norm outlier tokens are not salient in language modeling. Thus, the register tokens do not have impacts on long text comprehension.

c) Experimental comparision between register and corner tokens:

Based on the results of Table 2 and Table 3 of the paper [1], Register tokens may not necessarily make CLIP performance improvment. It indicates that learnable tokens are not the key to improve short-text understanding of CLIP.

*All results are derived from the Table 2 and Table 3 of paper [1]. [1] Vision Transformers Need Registers

Meanwhile, register tokens are also not the key to improve the long-text understanding of CLIP. As shown in the table below, we directly incorperate register tokens into text encoder on CC3M+short text and CC3M+long text, respectively. It shows incorporating register tokens in the text encoder does not yield improvements in averaged performance on both long-text-image and short-text-image retrieval tasks. Instead, our method boosts baseline by 2.05% and 0.18%, respectively, when trained with and without long texts.

Q17: Insights behind the improvements in long texts (Q3)

The corner tokens can facilitate [CLS] token in aggregating the diverse textual information within long texts to enhance the long text understanding ability of model. Compared to long texts, short texts have less textual information, where the corner tokens are hard to play a role. Thus, there are less improvement from corner tokens on short texts.

Q22: Are there any potential surrogate tasks for assessing understanding in conjunction with long texts and images?

This is an really interesting question. Recent works [7,8], which consider connecting images with long texts, also use long-text-image retrieval to assess the alignment between long texts and images. Beyond retrieval tasks, we believe there are other tasks to assess the benefits from the long text-image alignment, e.g., image understanding with MLLM.

Concretely, we finetune the pre-trained CLIP from openai with the proposed method, where the image and text encoder are unlocked. Then we incorperate the finetuned image encoder with LLM and train the MLLM following LLaVA-1.5. The MLLM is evaluated on two vision-centric VQA benchmarks [9], i.e., MMVP [10] and RealWorldQA [11], as shown in following table. The results indicate that using long texts for constrastive learning enhances the image encoder's ability to extract accurate and comprehensive visual features, which leading to improved image understanding by Multimodal Language Models (MLLMs).

[7] Long-CLIP: Unlocking the Long-Text Capability of CLIP. 2024.

[8] MATE: Meet At The Embedding -- Connecting Images with Long Texts. 2024.

[9] Cambrian-1: A Fully Open, Vision-CentricExploration of Multimodal LLMs. 2024.

[10] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. 2024.

[11] Grok-1.5 Vision Preview. 2024.

Q18: Limited technical innovation of the generated long texts. The pipeline simply relies on standard MLLM inference for generating long texts.

Thanks for your valuable suggestion. The Dora dataset is introduced to fill the need of large-scale long text-image pair dataset in multi-modal learning field. Existing text-image pair datasets typically consist of short text, restricting the ability of trained models on processing long texts. As far as we known, Dora is the largest dataset consisting long texts for multi-modal learning. We believe the community will benefit from Dora dataset in future research. In future, we will use some alleviating hallucination methods (e.g., OPERA [2]) to improve the quality of long captions, which can further improve the dataset.

[2] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. 2024.

Q19: Are the corner tokens defined as separate 'text' elements rather than as learnable token parameters?

No, the corner tokens are, in fact, learnable token embeddings. In the NLP field, new tokens are typically added to the vocabulary of tokenizer rather than directly initialized as learnable embeddings, e.g., [CLS] and [MASK] token in BERT [3], and "vokens" in MiniGPT-5 [4] and DreamLLM [5].

Concretely, before feeding a text into attention blocks of transformer, a tokenizer is used to convert subwords (text tokens and special tokens, e.g., [CLS] token) into indices, based on their order in the vocabulary of tokenizer [6], e.g., [CLS] token is coverted to 101 by BERT tokenizer. Then, these indices are converted to learnable token embeddings with a lookup table that stores embeddings [6]. In LotCLIP, we extend the vocabulary of tokenizer by adding corner tokens and update the lookup table. In this way, each corner token is converted to a learnable embedding before being fed into attention blocks.

[3] BERT: Pre-training of deep bidirectional transformers for language understanding. 2018.

[4] MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens. 2023.

[5] DreamLLM: Synergistic Multimodal Comprehension and Creation. 2023.

[6] Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. 2016.

Q20: Are corner tokens still effective without the BERT's CLS token?

If the [CLS] token is removed, corner tokens can also be used to represent texts, because corner token is varient of [CLS] token. As shown in the table below, the performance of using averaged features from corner tokens only degrades using features from the [CLS] token by 0.49% on average.

Q21: Ablates the introduction of LotCLIP, replacing the BERT text encoder with an OpenAI-style text transformer

Thanks for you suggestion. The experimental results of using OpenAI-style text transformer are shown in the Table 4 of the main manuscript, and we also present the results in the following table. CLIP's text encoder disables non-causal attention masks and uses the feature of last token as text feature. For fair comparison, R-LotCLIP adopts the same text encoder architecture. The results demonstrate the effectiveness of LotCLIP with using OpenAI-style text transformer as text encoder.

From the additional clarifications and experiments, the proposed corner token for the text encoder appears to be both innovative and more effective than the register token scheme used in the visual encoder, particularly in processing long captions. I believe the authors' presentation in the rebuttal has convincingly demonstrated its strength, leaving me with no remaining questions. Therefore, I no longer support rejecting it (score: 3 to 5).

Meanwhile, I view the 'register token' scheme as a crucial baseline for emphasizing the novelty and superiority of the proposed corner token, since both introduce additional learnable tokens into the pre-trained model. However, because the discussion about the register token was omitted in the initial draft, I am concerned that incorporating all discussions related to the register token, which have so far been addressed in the rebuttal, might require a significant revision.

Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.

Q1: Lack of details for addressing the limitation for the token length limitation of the text encoder in Sec. 3.4.

Sorry for confusion. We set a token length limitation of 128 for the text encoder based on the following two considerations:

• Token length of training text. For pre-trained models, the token length limitation can be arbitrarily set to any positive integer value. As shown in Figure 4 of Sec. 3.4, the results indicate that a text input limitation of 77 tokens is insufficient for model that requires long text comprehension. However, the training dataset has average of 136 tokens per text, and larger token length limitation may not bring more information.
• Balance of training efficiency and performance.: In Figure 4 of Sec. 3.4, a smaller token number may lead to performance degradation due to insufficient encoding of text information, while a larger token number may increase computational complexity. Thus, to balance the training efficiency and performance, we generally choose 128 as the maximum token length.

We will add more details in updated version.

Q2: For the attention mask part, if the corner tokens can be seen by other text tokens, how does it perform? In Table 3, how the model will the model perform if the corner taken is removed?

Thanks for the valuable suggestion. We have added the following experiments:

a) Performance is not improved (68.19 v.s. 68.20) when the corner tokens can be seen by other text tokens. This is because allowing corner tokens to be seen by text tokens can influence the diversity of aggregated text features, leading to worse performance.

b) Removing the corner token degrades LotCLIP's average performance by 1.98%, which further demonstrates the effectiveness of corner tokens.

We will include the results in Table 3 of the revision.

Q3: The year of references for NeurIPS papers e.g., [10] and [14], is not correct.

We apologize for any previous errors in the references, which we will carefully verified and rectified in the revised paper.

Incorrect Reference in paper:

[10] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024

[14] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. Uniclip: Unified framework for contrastive language-image pre-training. Advances in Neural Information Processing Systems, 35:1008–1019, 2022

[25] Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.

Rectification:

[10] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. Adv. Neural Inform. Process. Syst., 36:35544–35575, 2023

[14] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. UniCLIP: Unified framework for contrastive language-image pre-training. Adv. Neural Inform. Process. Syst., 35:1008–1019, 2022

[25] Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. StableRep: Synthetic images from text-to-image models make strong visual representation learners. Adv. Neural Inform. Process. Syst., 36:48382–48402, 2023

Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.

Q1: LotCLIP on other similar algorithms, *e.g.* ALIGN, CoCa.

Our LotCLIP can be applied on other similar algorithms as well, e.g. ALIGN, CoCa. The results on CC3M are presented in the table below, which demonstrate that LotCLIP also enhances the long text comprhension ability of other similar algorithm besides CLIP.

[1] CoCa: Contrastive Captioners are Image-Text Foundation Models

Q2: Illustrate learning process of corner tokens

Thanks for the valuable suggestion. The corner tokens are learnable tokens, which are placed before text tokens and after [CLS] token. We demonstrate the training process of corner tokens in below.

Inputs:

• image_encoder
• text_encoder
• tokenizer
• image_processor
• learnable corner tokens: $[`Cor 1`], [`Cor 2`],\cdots, [`Cor m`]$
• minibatch of long texts: $T_{\text{long}}$ [$n,l$]
• minibatch of short texts: $T_{\text{short}}$ [$n,l$]
• minibatch of images: $I$ [$n,h,w,c$]
• learned temperature parameter: $t$

Training Process:

# The corner tokens are placed in front of text before the tokenization process.

# After tokenization, the first token of text input is [CLS] token,

# while the i-th token (1<i<=$m$+1) is $[`Cor i-1`]$ token.

$T_{\text{long}}$ $=$ tokenizer($[`Cor 1`], [`Cor 2`],\cdots, [`Cor m`]$+ $T_{\text{long}}$)

$T_{\text{short}}$ $=$ tokenizer($[`Cor 1`], [`Cor 2`],\cdots, [`Cor m`]$ + $T_{\text{short}}$)

# Image pre-processing $I$ $=$ image_processor($I$)

# Build attention mask mechanism $\mathcal{A}$ $=$ np.ones([$l$,$l$])

# Text tokens do not attend to corner tokens $\mathcal{A}[m+1:, 1:m+1]$ $*=$ 0

# [CLS] token and corner tokens do not interact with each other $\mathcal{A}[:m+1, :m+1]$ $=$ np.eye(m+1)

# The attention mask $\mathcal{A}$ is multiplied by original attention mask used in the text encoder

# to eliminate the influence from [PAD] tokens to [CLS] and text tokens.

$\mathcal{A_{\text{long}}}$ $=$ $\mathcal{A}$ $\cdot$ text_encoder.build_attn_mask($T_{\text{long}}$)

$\mathcal{A_{\text{short}}}$ $=$ $\mathcal{A}$ $\cdot$ text_encoder.build_attn_mask($T_{\text{short}}$)

# Extract text features.

# The attention mask controls the interaction among tokens within in each attention block.

# The text_encoder outputs the features of the first $m+1$ tokens, where the first

# one is [CLS] token and the remains are corner tokens.

$f_{lt}$ $=$ text_encoder($T_{\text{long}}$, attention_mask $=$ $\mathcal{A_{\text{long}}}$) # [$n,m+1,d$]

$f_{st}$ $=$ text_encoder($T_{\text{short}}$, attention_mask $=\mathcal{A_{\text{short}}}$) # [$n,m+1,d$]

# Extract image features.

$f_\text{i}$ $=$ image_encoder($I$) # [$n,d$]

# Normalization

$f_{lt}$ $=$ l2_normalize($f_{lt}$, axis=-1)

$f_{st}$ $=$ l2_normalize($f_{st}$, axis=-1)

$f_{i}$ $=$ l2_normalize($f_{i}$, axis=-1)

labels $=$ np.arange(n)

# Loss computation

$logits_{\text{short}}$ $=$np.dot($f_{i}$, $f_{st}$[:,0,:]) $\cdot$ np.exp($t$)

$loss_{\text{short}}^{i2t}$ $=$ cross_entropy_loss($logits_{\text{short}}$, labels, axis=0)

$loss_{\text{short}}^{t2i}$ $=$ cross_entropy_loss($logits_{\text{short}}$, labels, axis=1)

$loss_{\text{short}}$ $=$ ($loss_{\text{short}}^{i2t}$+$loss_{\text{short}}^{t2i}$)/2

$loss_{\text{long}}$ $=$ 0

For $k \in [0, m]$:

      $logits_{\text{long}\_k}$ $=$ np.dot($f_\text{image}$, $f_\text{long}$[:,$k$,:]) * np.exp($t$)

      $loss_{\text{long}\_k}^{i2t}$ $=$ cross_entropy_loss($logits_{\text{long}\_k}$, labels, axis=0)

      $loss_{\text{long}\_k}^{t2i}$ $=$ cross_entropy_loss($logits_{\text{long}\_k}$, labels, axis=1)

      $loss_{\text{long}}$ $+=$ ($loss_{\text{long}\_k}^{i2t}$+$loss_{\text{long}\_k}^{t2i}$)/2

loss $=$ $loss_\text{short}$ + $loss_{\text{long}}$

Q3: Should have also reported results on ALIGN

Thanks for your valuable suggestion. We report the performance of ALIGN pre-trained on COYO-700M in the table below. Even trained with smaller scale dataset, LotCLIP outperforms ALIGN on long-text-image retrieval task. We will include the results of ALIGN for comparision in the revised version of our paper.

Q4: Not correct/incomplete references

We apologize for any previous errors in the references, which we will carefully verify and rectify in the revised paper.

Q5: Diagram on corner token learning process

Please refer to our reply to Q2.

Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Bellow we address your concern separately.

Q1: Difference between Dora dataset and DreamLIP.

The main difference between Dora dataset and the dataset proposed in DreamLIP lies on two aspects:

• Larger Dataset Scale: Our dataset has 8x image volume and 5x text volume of DreamLIP as shown in following table.
• Longer Token Length: Our dataset has average of 136 tokens per text, which is twice as many as DreamLIP (136 v.s. 75).

It is mentioned that the long texts of DreamLIP are splitted into short texts and sperately utilized. In this way, the usage text of DreamLIP are tokenized to around 21 tokens on average, which not exploring how to make text encoders better understand long texts.

[1] We report the statistical results derived from 15 million data released by DreamLIP.

Q2: More details on how the Dora dataset has been built will be useful.

Thanks for your valuable suggestion. We use multi-modal large language models (MLLMs) to generate long texts by prompting 'Please describe the image in details'. To prevent the bias introduced by MLLMs' hallucinations, three kinds of MLLMs are introduced to generate the diverse long texts from different model knowledges. We also provide the hyper-parameter settings of the used MLLMs:

Q3: Statistics of Dora dataset, comparing with the other datasets analyzed in table 1 of the main manuscript.

As shown in the table below, we report some statistics of our dataset. To the best of our knowledge, Dora is the largest dataset consisting long texts for multi-modal learning. And we are continuing expanding the size of Dora by integrating additional MLLMs for long text generation, as well as gathering more publicly available datasets.