We appreciate your positive feedback on the architecture design and paper logic. Below, we address each of the noted weaknesses and questions in turn.

W1/Q2 - Duration-Proportional Information Assumption and practice

The Duration‑Proportional Information Assumption is motivated by three observations:

• Video information content grows as frame rate increases.
• However, the rate of that growth diminishes at higher frame rates.
• As the frame rate approaches infinity, the total information converges to a finite upper bound. (Equation 1)

The traditional tokenizers uses a fixed spatiotemporal compression rate, which implicitly assumes a linear, unbounded increase in information with frame rate. In contrast, VFRTok instead employs a fixed-length latent sequence to approximate that theoretical upper bound.

In the table below, we compare reconstruction performance on the Adobe240fps dataset between standard tokenizers and VFRTok across multiple frame rates. As expected, the traditional tokenizers' token count and its reconstruction quality grows linearly with frame rate, implying that it overallocates capacity at high frame rates. While VFRTok's reconstruction quality degrades modestly at higher frame rates. The experimental result confirms that real video information scales sublinearly. We note that VFRTok was trained primarily on 24fps and 30fps data, with only 22 clips at 120fps; increasing the proportion of high–frame‑rate examples should further improve its fidelity at those rates.

While neither the Frame‑Proportional nor the Duration‑Proportional Information Assumption perfectly captures the sublinear growth of video information with frame rate, the Duration‑Proportional Assumption nonetheless offers clear practical advantages over a purely Frame‑Proportional approach in downstream generation. By fixing the latent length, it enables a single DiT model to denoise and generate videos at arbitrary frame rates, and it delivers better efficiency gains as target frame rate increases.

W2 - Experimental details

We will add more experimental details in the appendix in the next version. The configuration of VFRTok and LightningDiT are shown in the following tables.

Q1 - Advantages compared to existing works

Although prior methods also train on raw videos at multiple frame rates, their strategies differ fundamentally from VFRTok. They either resample every video to a single, fixed frame rate before encoding or simply ignore frame rate by feeding all frames into the model. The former approach constrains the model to that chosen frame rate. While the latter enforces a uniform spatiotemporal compression rate regardless of frame rate, forcing high-frame-rate videos to be “slowed down” into unnecessarily many tokens and low-frame-rate videos to be “speed up”. Both of which limit the expressivity of the latent representation.

As shown in Figure 1, these methods operate under a Frame‑Proportional Information Assumption, where token count grows linearly with frame rate (e.g., a 120 fps clip demands four times the tokens of a 30 fps clip). By contrast, we find that a video’s information content increases sublinearly with frame rate and asymptotically approaches an upper bound (Duration-Proportional Information Assumption). VFRTok leverages this insight by using a fixed number of latent tokens to approximate that bound, thus unifying videos of the same duration but different frame rates under one consistent representation. Since the downstream DiT model’s computational complexity scales geometrically with the number of latent tokens, VFRTok’s constant-length encoding delivers significant efficiency gains for high-frame-rate video generation.

Q3 - Applicability

When contrasting the Duration‑Proportional Information Assumption with the Frame‑Proportional Information Assumption, we are claiming that allocating information uniformly over time yields statistically better performance than allocating it uniformly over frames. However, at the sample level, both assumptions share a common limitation: videos exhibit widely varying content complexity. Specifically, videos vary in semantic, textural, and motion complexity, and thus should require varying information capacity. In conventional video codecs (e.g., H.264), achieving the same reconstruction quality for two videos of differing complexity typically demands different bitrates. Analogously for tokenizers, the length of the latent representation ought to adapt to a video’s content complexity. This content‑adaptive modeling perspective is orthogonal to the hypothesis explored in our paper.

While content‑adaptive tokenizers have been explored in a few works, the key challenge lies in enabling the downstream DiT model to support estimating a video’s complexity from its caption alone. We plan to address this in future work.

Q4 - Symmetric training

We train a VFRTok variant under a symmetric encoding regime. Specifically, we initialized from the K600 pre-trained model in stage 1 for quick verification. In stage 2, which uses both K600 and BVI-HFR datasets, we disabled asymmetric encoding, forcing both encoder and decoder to operate at the same frame rate. As shown in the following table, this symmetric training yields a marginally better reconstruction in the symmetric setting but suffers a pronounced degradation under asymmetric encoding. Note that our stage 1 initialization itself used asymmetric frame rates, which partially narrows the gap in asymmetric reconstruction. Overall, these results confirm that asymmetric training is crucial for enabling VFRTok to generate videos at arbitrary frame rates.

Q5 - Partial RoPE and generation consistency

Our experiments show that adding Partial RoPE does not compromise temporal consistency. As reported in the following table, we compute the average L2 distance between consecutive frames with and without Partial RoPE. The results reveal only a negligible difference, confirming that Partial RoPE maintains temporal coherence.

Meanwhile, we find that incorporating Partial RoPE yields markedly more coherent motion in the generated videos. In particular, with it enabled, objects and human figures exhibit substantially less fragmentation, deformation, and distortion throughout their movements.

We appreciate your positive feedback on our motivation and architectural design. Thank you as well for illustrating the application scenario of game video generation. Below, we address each of the noted weaknesses and questions in turn.

W1 - Duration-Proportional Information Assumption and content-adaptive modeling

We appreciate your concerns. It is true that fast‑moving videos often convey more information. However, it is important to note that high frame‑rate video and high‑speed motion are not synonymous: a video’s frame rate and the speed of the motion it depicts are two decoupled factors. Consequently, traditional video tokenizers, which hold the Frame-Proportional Information Assumption, face the same limitation. Given a fixed frame rate, a standard tokenizer allocates a fixed number of tokens to represent each frame (often far more than VFRTok). As motion increases, reconstruction quality naturally degrades under this scheme.

More generally, videos vary in semantic, textural, and motion complexity, and thus should require varying information capacity. In conventional video codecs (e.g., H.264), achieving the same reconstruction quality for two videos of differing complexity typically demands different bitrates. Analogously for tokenizers, the length of the latent representation ought to adapt to a video’s content complexity. This content‑adaptive modeling perspective is orthogonal to the hypothesis explored in our paper.

While content‑adaptive tokenizers have been explored in a few works, the key challenge lies in enabling the downstream DiT model to support estimating a video’s complexity from its caption alone. We plan to address this in future work.

Q1 - Fast motion modeling

VFRTok’s latent capacity is adequate for capturing rapid motion. As illustrated in Figure8, VFRTok successfully generates scenes of fast-paced activities, such as skiing and pull‑ups. Notably, the examples shown in Figure~8 are drawn from clips only 2–3 seconds in length, underscoring the model’s ability to represent and recreate high‑speed motion faithfully.

Q2: Partial RoPE enhances motion modeling

Our analysis shows that incorporating Partial RoPE yields markedly more coherent motion. In particular, with Partial RoPE enabled, objects and human figures exhibit substantially less fragmentation, deformation, and distortion throughout their movements.

We appreciate your positive feedback on the technical novelty and experimental results. Below, we address each of the noted weaknesses and questions in turn.

W1 - Additional captions

We will further refine captions to enhance readability. For example, we may modify the caption of Figure 1 to “The number of tokens for other tokenizers grows with frame rate. VFRTok maintains a fixed number of latent tokens tied to video duration and supports asymmetric frame‑rate encoding and decoding.”

W2 - Experimental details

We will expand on the details of the training.

Inference details. During inference, VFRTok supports decoding videos of any frame rate from latent tokens denoised by DiT. Specifically, the VFRTok decoder calculates the number of learnable tokens based on the incoming frame rate and injects the timestamp-based RoPE into each token.

Learnable token details. We adopted the same design as existing image 1D tokenizers: on the encoder side, we learn N independent latent tokens, whereas on the decoder side, we use a single shared token. Although we experimented with replacing the decoder’s shared token with a fixed-length set of independent tokens, which improves reconstruction fidelity, this change eliminated the decoder’s ability to flexibly handle variable‑frame‑rate decoding.

W3 - Quantitative results for video interpolation

The following table shows quantitative video‑interpolation results comparing FLAVR and VFRTok. Although FLAVR is a strong video‑interpolation baseline and outperforms VFRTok on this task, VFRTok was not designed primarily for interpolation. First, VFRTok employs an extremely high compression rate to enable efficient video generation, creating a tighter bottleneck than dedicated interpolation models. Second, our training set contains only 22 clips at 120fps and no 60fps videos, whereas interpolation models are typically trained on large-scale, high‑frame‑rate data. In summary, while VFRTok can perform interpolation, its principal application remains general video generation.

Q1 - High frame rate video generation

Traditional Frame-Proportional Information Assumption tokenizers must encode high frame rate videos during DiT training to enable high frame rate generation. In contrast, VFRTok’s Duration‑Proportional Information Assumption allows it to map videos of any frame rate into fixed‑length latent sequences. Because the VFRTok decoder converts these fixed‑length latents into variable‑length patch tokens, it can efficiently generate high frame rate videos, even when the DiT model is trained exclusively on low frame rate data.

Q2 - Release model weights

We plan to release both the code and model weights concurrent with the camera‑ready version.

Q3 - Position embedding for latent tokens

We adopt the architecture of existing 1D image tokenizers by incorporating learnable absolute positional encodings into our latent tokens. We omit RoPE because modeling relative positions between latent tokens is unnecessary. Unlike video patch tokens, where proximity in time and space implies similar content, the latent tokens aggregate global information, so adjacent latents need not correspond to semantically related features.

We appreciate your positive evaluation of our work’s motivation, design strategy, and experimental findings. Below, we address each of the noted weaknesses and questions in turn.

W1 - RoPE related works

We will include more comprehensive references to 3D RoPE and NoPE in the camera‑ready manuscript. It is important to note that existing 3D RoPE follows the same channel‑division strategy as VFRTok: each attention head’s channels are split into three groups, with RoPE applied independently along the temporal, width, and height dimensions (Equation 2). To our knowledge, no other work employs this split-head Partial RoPE approach.

W2/Q1 - Generalization to resolution

Extending VFRTok to handle variable resolutions is another direction we are exploring. A natural extension of our Duration‑Proportional Information Assumption is a Spatial‑Temporal‑Proportional Information Assumption, which would allow videos of differing frame rates and resolutions to be encoded into fixed‑length latents. In practice, we would achieve this simply by applying interpolated RoPE along the spatial dimensions (height and width) for different‑resolution inputs.

That said, stretching the video representation across three dimensions may lead to capacity inefficiencies in low‑frame‑rate and low‑resolution segments. To mitigate this, we plan to design a sublinear, variable‑compression‑rate video tokenizer in future work.