Response to Reviewer 4

We thank the reviewer for their thoughtful comments, and for highlighting the importance of grounding our assumptions, clarifying applicability, and assessing computational overhead. We address each of your concerns below.

On Access to Relative Transformations

We respectfully disagree with the claim that requiring access to relative transformations (actions) is a “major limitation” of our method. Our design is both consistent with prior work and motivated by embodied learning principles.

First, the assumption of known or controlled transformations is standard in the equivariant SSL literature. Many works in this space assume access to transformation/action parameters [A–C] or to at least two pairs of views where the same transformation/action is applied [D, E]. Our setting is consistent with this standard practice in the literature and follows this precedent by modeling the transition $ p(z_{t+1} \mid z_t, a_t), where \a_t$ is either known or can be approximated.

Second, as noted in the Introduction (see paragraph 3 in Intro), our design is grounded in the embodied learning perspective, where perception is inherently active. In such cases (e.g., robotic manipulation, foveated vision, embodied agents in simulation) the agent performs the actions itself and therefore has access to them. From a neuroscience perspective, this setup aligns with the concept of efference copy (or corollary discharge), i.e., an internal copy of motor commands sent to both the motor system and sensory areas to anticipate the consequences of self-generated actions. This mechanism enables animals to predict sensory inputs conditioned on their own actions, and we adopt the same inductive bias in seq-JEPA.

In domains where the agent does not execute the actions itself (e.g., passive observation or multi-agent settings), estimating the actions becomes necessary. In this case, the model can either be given approximate action estimates (e.g., via egomotion sensors or inverse modeling), or extended to jointly infer latent actions along with representations by observing other agents. This opens a natural direction for future work; building theory-of-mind-capable agents that model others’ actions and policies from observational data.

On Dataset Scope and Practical Applicability

While 3DIEBench involves synthetic renderings, our experiments are not limited to artificial domains. We also evaluate seq-JEPA on CIFAR-100, TinyImageNet, and PLS (Predictive Learning across Saccades on STL-10), all of which use natural images. In these settings, the “action” is either:

• The relative augmentation parameters (e.g., crop coordinates, jitter levels, blur kernel size), which are available in any augmentation pipeline.

• The relative positions between foveated glances, sampled with biologically plausible saliency and inhibition-of-return (IoR) heuristics (in PLS), which simulate human-like visual behavior.

Our results on these settings confirm that our approach generalizes beyond synthetic 3D datasets, and operates effectively in low-resolution, weakly-structured, or biologically grounded regimes.

On Separability and Interpretation of Learned Representations

We appreciate the request for a closer analysis of the learned equivariant and invariant features. Our probe-based results in Table 1 already reveal this division: the encoder (pre-aggregation) preserves action-relevant information (high R² in transformation prediction), while the aggregator (post-aggregation) supports invariant object classification (high top-1 accuracy).

Additionally, we include UMAP visualizations in Figure 5, which show this duality in latent space. The left and middle panels display structured orbits of transformed views around class centers in the encoder space—indicating equivariance. The right panel, which shows post-aggregation representations, demonstrates class-wise collapse and increased inter-class separability, confirming that the aggregator learns to filter transformation-specific variation. Finally, the middle panel, colored by rotation angle, confirms that encoder orbits are transformation-aware: latent trajectories evolve smoothly with rotation, and are consistent across object instances (e.g., compare the red cluster in the left panel with the corresponding cluster in the middle panel).

On Compute and Memory Overhead

We report pretraining compute time across methods (see table below). All experiments are run under the same configuration: a single A100 GPU, 128×128 resolution, batch size 512, for 1000 epochs on 3DIEBench. seq-JEPA with a sequence length of one has similar runtime to other baselines (e.g., BYOL, SimCLR), while seq-JEPA with sequence length of 3 incurs a moderate increase in wall-clock time (15.1 GPU-hours) due to encoding additional action-view pairs.

GPU-Hour Comparison Table

seq-JEPA’s inference cost is primarily governed by two factors: sequence length and input resolution. Below, we report detailed FLOP counts (in Gigaflops) per forward pass of a single datapoint across a range of configurations (Note: these reflect inference-time cost):

FLOPs by Configuration

We can observe that:

• For single-view inference at 128×128 resolution, the encoder requires only 0.60G FLOPs, which matches the compute cost of standard SSL baselines such as SimCLR and BYOL.

• As sequence length increases, the additional compute cost grows sublinearly as additional views should be encoded and aggregated. For example, lengths 2 and 3—where invariant performance improves substantially—incur ~1.2G and ~1.85G FLOPs, respectively, remaining within practical limits for most modern applications.

• Crucially, this cost can be offset by reducing input resolution. In our saccade-based setup, we can feed low-resolution foveal glimpses (e.g., 64×64 or 84×84 crops sampled via saliency maps), which require only 0.15G–0.30G FLOPs per view. This enables the use of longer sequences at an overall compute cost comparable to full-resolution, single-view pipelines (e.g. compare the flops of aggregating eight 64 by 46 crops with flops of encoding a single 224 by 224 image in the second row of the table). This approach is particularly appealing in real-time robotic systems, where sensors frequently provide partial, asynchronous, or spatially limited observations (e.g., narrow field-of-view cameras, event-based sensors, or sequential glances across a scene). In such contexts, seq-JEPA can accumulate and integrate these glimpses over time to form a compact, invariant summary for downstream reasoning, while maintaining low computational cost.

We thank the reviewer again for their helpful suggestions. We hope that these clarifications, particularly around access to actions, applicability to natural images, and compute-performance trade-offs, address your concerns. We also invite you to consult our responses to the other reviewers.

References

[A] Garrido, Quentin, Laurent Najman, and Yann LeCun. Self-supervised learning of split invariant equivariant representations. Proceedings of the 40th International Conference on Machine Learning. 2023.
[B] Garrido, Quentin, et al. Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504 (2024).
[C] Devillers, Alexandre, and Mathieu Lefort. EquiMod: An Equivariance Module to Improve Visual Instance Discrimination. The Eleventh International Conference on Learning Representations, 2023.
[D] Yerxa, Thomas, et al. Contrastive-equivariant self-supervised learning improves alignment with primate visual area IT. NeurIPS 2024.
[E] Shakerinava, Mehran, Arnab Kumar Mondal, and Siamak Ravanbakhsh. Structuring representations using group invariants. NeurIPS 2022.

We thank the reviewer for their thoughtful and detailed review. Below, we address your questions and concerns.

On Substantiating Equivariance Beyond Transformation Prediction

We agree that demonstrating full functional equivariance—where the representation transforms in a predictable manner under input transformations—is stronger than transformation regression alone. Regression is often used since functional equivariance is hard to quantify in high dimensions. Therefore, our evaluation follows the established practice in equivariant SSL, where transformation prediction is used as a practical proxy for equivariance [B–D].

To go beyond this, we also provided qualitative evidence in Fig 5 (left and mid panels), which shows that UMAP projections of encoder representations form smooth, structured orbits around object category centers, with a continuous gradient across the rotation angle (see the visible alignment between the red band in class-based coloring and its smooth rotation-based coloring in Fig 5). This behavior is consistent with a structured, transformation-aware latent geometry and offers indirect evidence of equivariance.

That said, we agree that direct testing (e.g., using linear equivariant maps or commutativity checks with group actions) would provide more rigorous validation. We consider this a valuable direction for follow-up work, especially in the context of latent world modeling where group structures may be implicit, partial, or noisy.

Clarifying Transformation Sequence Generation

In our current setup, transformation sequences are externally defined and fixed during training—there is no learned policy over actions. Specifically:

• 3DIEBench: Each object is rendered with a random sequence of 3D rotations, sampled uniformly from $(-\pi/2, \pi/2)$ across axes. These sequences are independent and diverse.

• PLS: Saccade trajectories are generated using saliency maps with inhibition-of-return (IoR), producing structured, non-repetitive sequences over natural STL-10 images. This simulates foveated active vision and enables real-world applicability.

• CIFAR-100 / TinyImageNet: Sequences are composed from standard data augmentations (crop, jitter, blur) with known parameters. “Actions” are defined as the difference between consecutive augmentation parameters.

Thus, our framework handles both randomized and structured transformation regimes.

On the SIE Performance Drop

Thank you for flagging this. The reported SIE performance drop is due to differences in hyperparameter regimes. The original SIE paper trained with: 2000 epochs, 256×256 input resolution, and batch size 1024.

In contrast, as noted in Section 3 of our paper, we standardized training across all baselines (including SIE) to: 1000 epochs, 128×128 resolution, and batch size 512. This was necessary to enable a fair comparison across baselines within our compute budget.

Comparison to [A]

We thank the reviewer for suggesting this comparison. [A] proposes a regularization that constrains adjacent views (e.g., with small pose deltas) to lie on smooth trajectories in latent space. This reflects a “local linearity” prior, which is well-motivated for passive or smooth environments (e.g., robotic arms, camera pans).

In contrast, seq-JEPA is designed for active perception settings—where transitions may be sparse, discontinuous, or highly non-local (e.g., long-range saccades, or abrupt object rotations). It assumes no trajectory smoothness or small-angle assumptions.

We empirically compare [A] by adding it as a baseline across three datasets. We implemented their best-performing model variant (VICReg + trajectory regularization) and tuned the regularization strength $\alpha$ on 3DIEBench across values {0.001, 0.01, 0.1, 1.0}. We found $\alpha = 0.01$ to yield the highest top-1 accuracy and rotation prediction and used it for the other two datasets. Results are given below:

We see that trajectory regularization indeed improves over VICReg (compare with tables 1 and 2 in the paper), which shows that imposing geometric priors can improve both invariant and equivariant performance (even when the priors do not fully materialize in the environment, e.g., when we have non-smooth angle changes as in 3DIEBench).

Unseen Transformation Generalization (OOD Rotations)

Since 3DIEBench does not include an OOD test set, we created one ourselves to enable evaluation on unseen transformations. The original dataset samples rotations from $(-\pi/2, \pi/2)$. Our OOD test set instead uses the disjoint range $(-\pi, -\pi/2) \cup (\pi/2, \pi)$, ensuring no angular overlap with the training set.

This makes the task strictly extrapolative, unlike [A], which evaluates models on denser but still interpolative pose samples from within the same angular support.

Rendering followed SIE’s protocol and took ~5 hours on a single A6000 GPU. We will release this benchmark publicly.

From the table below, we see that all methods fail at OOD rotation decoding (with R²s even reaching near −1.0). The sharp drop in R² values may be attributed to a domain shift in transformation geometry: while representations are well-aligned with ID rotation trajectories (inside the range $(-\pi/2, \pi/2)), they are not geometrically structured to extrapolate beyond this range to \(- \pi, - \pi/2) \cup (\pi/2, \pi)$. The near −1.0 R² values likely result from a reversal in regression sign: the learned probe fails to interpret latent geometry correctly in the unseen region and predicts anti-correlated values. This supports our interpretation that the latent space does not generalize to novel angular regions, despite appearing structured within the ID range.

Despite the failure in OOD equivariance generalization, seq-JEPA exhibits a graceful degradation in invariant classification accuracy compared to other baselines. We hypothesize that this robustness stems from its sequence-level aggregation mechanism: even when latent representations become less equivariant under OOD transformations, aggregation across multiple views still filters out transformation-specific variability and recovers shared semantic content. Thus, seq-JEPA maintains object identity better under distribution shift, indicating that aggregation over input views promotes invariance, even when equivariance is imperfect.

Clarifying MLP vs. Transformer Projectors

Training with a transformer projector (for models originally with an MLP projector) was designed to isolate the contribution of model capacity by controlling for architectural differences in the projection head. Specifically, we tested whether transformer-based projectors alone account for any performance gains. We did not see any benefit from switching to transformer projectors in any of the baselines and the results in Tables 1 and 2 reflect original MLP-based models. We will clarify this in the paper and include the full transformer-projector results in the appendix. Below, you can see these results for the baselines on 3DIEBench (plus sign indicates a transformer projector).

Time and Memory Comparison

We kindly refer the reviewer to our FLOP and GPU-hour comparisons in the response to Reviewer 4. In summary, seq-JEPA maintains reasonable compute overhead (e.g., 12.3G–15.1G GPU-hours vs. 11–12.5G for baselines), and can amortize sequence cost by using low-resolution glimpses (e.g., a sequence of small foveated crops from a large image), which are viable in embodied vision (as exemplified in our PLS setup, where low-resolution glimpses support effective sequential aggregation).

Refs:

[A] Pose-aware Self-supervised Learning with Viewpoint Trajectory Regularization [B] Self-supervised Learning of Split Invariant Equivariant Representations [C] Learning and Leveraging World Models in Visual Representation Learning [D] Contrastive-equivariant SSL improves alignment with primate visual area it

Response to Reviewer 2:

We thank the reviewer for their thoughtful and encouraging feedback. We are glad you found the problem setup well-defined, the architecture intuitive, and the evaluations thorough. Below, we address your main question and provide our intuition for how equivariant and invariant representations emerge in seq-JEPA—namely, why the encoder retains action information (equivariance) and how the aggregator yields invariant representations.

Why does seq-JEPA retain action information and develop equivariance?

We begin with some background. In contrastive SSL using InfoNCE, it has been shown that, under certain assumptions, the learned representations $z$ and $z'$ can recover the data-generating factors of observations $x$ and $x'$ [A]. However, this recovery only holds for factors shared across the pair, meaning factors that are invariant (e.g., object category). Transformation- or action-specific latents (e.g., pose or viewpoint) are, by design, not captured.

While non-contrastive predictive methods (e.g., BYOL, JEPA variants—including seq-JEPA) do not offer formal identifiability guarantees, they implicitly solve a latent-space world modeling task, aiming to model transitions of the form $p(z' \mid z, a)$. If the model does not have access to the action $a$, it must implicitly marginalize over it—leading to greater ambiguity and a broader solution space. In contrast, when $a$ (or a reliable estimate; see our response to Reviewer 4) is available, the encoder is incentivized to preserve action-relevant information. This makes the conditional prediction $p(z' \mid z, a)$ more tractable and structured (action-conditioning makes it easier to predict the next state).

Why does aggregation induce invariance?

Given the above, seq-JEPA exhibits two other emergent behaviors:

First, the aggregator naturally promotes invariance. It processes multiple transformed views (e.g., rotated object images) whose representations orbit around the category center in latent space (Figure 5, left panel). By attending to and averaging these, the aggregator filters out transformation-dependent components while reinforcing shared content—i.e., invariant factors. As a result, the aggregated representation gravitates toward the center of the class cluster, reducing intra-class variance and increasing inter-class separability in the aggregated latent space (see Figure 5, right panel).

Second, equivariance improves by observing longer sequences of action-observation pairs during training. Feeding additional tuples $(z_i, a_i, z_{i+1})$ into the transformer-based aggregator creates a richer context. This aggregated context, denoted $z_{\text{agg}, t}$, is passed to the predictor, which learns to jointly encode $z_{\text{agg}, t}$ and the next action $a_t$ in order to predict the future state $z_{t+1}$. With access to a longer sequence—i.e., more transitions in working memory and a richer context—the predictor becomes better at capturing how specific actions influence latent dynamics. This enables it to more accurately approximate the transition distribution $p(z_{t+1} \mid z_{\text{agg}, t}, a_t)$. Because accurate prediction requires the model to preserve and utilize information about $a_t$, the encoder is implicitly encouraged to learn structured, more equivariant representations. Empirically, we indeed observe that increasing the training sequence length leads to improved equivariance performance (see Figure 7, left panel).

Emergence vs. enforcement

We emphasize that the emergence of both invariant and equivariant representations in seq-JEPA is not enforced explicitly, but rather emerges from its predictive world modeling objective and inductive biases. Our discussion above clarifies how components like action conditioning, sequential aggregation, and predictive modeling each contribute to this separation.

Notably, the model is not constrained to learn this decomposition. In principle, it could converge to a trivial solution by marginalizing over actions and learning a fully invariant world model. However, this does not occur in practice. Instead, the design of seq-JEPA naturally encourages the encoder to retain transformation-sensitive information (to predict transitions), while the aggregator averages over such views (to form invariant summaries). We view this emergent disentanglement as a key strength of the method and a promising direction for further investigation.

Reference:

[A] Zimmermann, Roland S., et al. Contrastive learning inverts the data generating process. ICML 2021.

Response to Reviewer 1:

We thank the reviewer for their constructive comments and for highlighting the clarity and simplicity of our approach. We are pleased that you found the method easy to follow and appreciated the ablations and task selection. Below, we address your main concerns and clarify your question about the representation of actions.

Connection to [A]:

We thank the reviewer for pointing out the relevant work of Chavhan et al. [A], which indeed bears a conceptual connection to ours. In [A], the authors use an ensemble of heads trained on top of a base encoder (MoCo) to span a diverse spectrum of transformation sensitivities across the latent spaces of each head (encouraged via a diversity loss). Downstream probes then learn to linearly combine these feature heads depending on the desired invariance-equivariance trade-off.

While the motivation is related, seq‑JEPA offers a fundamentally different and more architectural approach. In our framework, equivariant and invariant representations emerge within a single model, with no need for additional heads or specialized loss functions. Specifically, the encoder produces equivariant representations, while the sequence aggregator (transformer) forms an invariant representation by aggregating views. This dual structure arises purely from our architectural inductive bias and predictive learning objective, which enables seq-JEPA to navigate the invariance-equivariance trade-off through its core architectural design, without requiring explicit regularization or auxiliary heads.

We will add a discussion of this connection in the final version to better position our work in the landscape of hybrid invariant-equivariant representation learning.

Clarification on Action Representation:

As noted in Section 2.2 of the paper, the relative transformations (actions) $a_i are defined as the transformation that maps \x_i to \x_{i+1}, i.e., \a_i := t_{i → i+1}$.

The specific instantiation of the action vector depends on the transformation setting. We have provided these details at the end of Appendix A, but we will clarify this more explicitly in the main text in the final version of the paper:

3DIEBench (Rotations): We use the 4D quaternion vector representing the relative rotation between two views, consistent with the original benchmark [B].

Predictive Learning Across Saccades: The action is a 2D vector indicating the normalized relative (x, y) position between the centers of two patches, simulating a saccadic eye movement.

Hand-Crafted Augmentations (Crop, Jitter, Blur): Each augmentation is parameterized as follows, and the corresponding action vector a_i is computed as the difference in parameters across views:

Crop: [x, y, height, width] (4D)

Color jitter: [brightness, contrast, saturation, hue] (4D)

Blur: [kernel standard deviation] (1D)

In all settings, the action vector is passed through a learnable linear projector (default: 128D), then concatenated to the visual representation before being passed to the transformer aggregator.

We again thank the reviewer for their thoughtful feedback and hope the above clarifications, along with the related responses to other reviewers, help further contextualize our design choices and contributions.

References:

[A] Chavhan, Ruchika, et al. Quality Diversity for Visual Pre-Training. ICCV 2023.

[B] Garrido, Quentin, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. arXiv preprint arXiv:2302.10283 (2023).