Visual Sync: Multi‑Camera Synchronization via Cross‑View Object Motion

a) Reliance on external models

While our pipeline integrates state-of-the-art tools such as VGGT, MAST3R, and CoTracker, our core contribution lies in the robust energy formulation and inference framework, which is modular and robust to upstream noise.

Our energy function value itself serves as a robust indicator for detecting unreliable outputs from preceding models. As shown in Tab. 4 (5th and bottom row) , our method leverages the energy landscape to actively filter out spurious pairs, allowing the system to tolerate failures in individual components—especially noisy pose predictions from VGGT.. This design ensures that localized errors (e.g., due to motion blur or egocentric viewpoints) do not compromise overall synchronization.

Crucially, our method does not require accurate poses for every frame. In practice, a subset of frames with reliable pose estimates is sufficient for robust synchronization, thanks to our energy-based optimization and filtering of spurious pairs. As a result, our method gracefully handles imperfect predictions and remains robust in challenging scenarios.

EgoHumans dataset presents severe challenges due to egocentric viewpoints and abrupt head motion (see Supp videos for visualization of these challenges). As shown in Tab. 3 (2nd and bottom row), using estimated poses on the challenging EgoHumans dataset yields a median error of 46.6 ms, close to the 28.6 ms oracle result with ground-truth poses.

We report camera pose and synchronization results (relative angular error in rotation and translation following VGGT) across randomly selected EgoHumans sports videos below, demonstrating the robustness of our approach under varying pose estimation qualities. Even with large rotation and translation errors in camera pose estimates, our method maintains low synchronization errors (e.g., 9.6 ms for Fencing and 19.4 ms for Tagging), indicating that pose noise does not directly translate to sync failure.

Note: High sync error in Tennis is due to distant camera placement, resulting in minimal observable motion (see supplementary video).

We further demonstrate our system's effectiveness by applying our method to challenging in-the-wild videos, as shown in the supplementary material (Supp Fig. 3 and Video). These include NBA basketball sequences and FIFA soccer sequences with noisy estimated camera poses. Our method still achieves good synchronization accuracy in these challenging real-world scenarios.

---

b) Computational cost and scalability

We have included a runtime and resource efficiency comparison against baselines such as MAST3R, Uni4D, and Sync-NeRF on the Panoptic dataset (15 videos, ~200 frames each) using a single A6000 GPU.

Our method is training-free, and most of the overhead comes from visual cue extraction. While we acknowledge that further optimizations (e.g., lightweight modules) can improve runtime, efficiency is not the primary focus of our work, which targets accuracy and robustness under challenging conditions. In our experiments, we show our system could accurate synchronize 30 cameras on the CMU Panoptic dataset. With additional computing resources and parallel video preprocessing, the overall running time can be significantly reduced.

In practice, our method is fast enough for typical multi-camera applications (e.g., 4-5 camera), such as movie production, sports analysis, and video surveillance, where offline processing is acceptable. We see further efficiency improvements (e.g., via lightweight models) as a valuable direction for future work, especially for very large-scale or latency-sensitive scenarios.

---

c) Failure case analysis

We provide detailed failure cases analysis of each module within our framework and qualitative examples in Supp.F. We will further improve the discussion of limitations and robustness of our method in future version.

a) Effective Integration of Recent Tools

We agree that the core idea—synchronized frames should obey epipolar constraints—is conceptually simple. However, we see this as a strength rather than a limitation. Synchronizing dynamic, unconstrained videos remains a highly challenging task, and a method that is both simple and effective is valuable in real-world settings.

While our method builds on recent advances such as VGGT, CoTracker, and MAST3R, our main contribution lies in the unified energy formulation and inference strategy that integrates both static and dynamic cues to solve a new problem these tools alone cannot address. We appreciate the reviewer’s recognition of our integration efforts, and believe that the clarity of our formulation is key to the method’s robustness across diverse settings.

---

b) Inconsistent frame rates

First, we would like to clarify that our method can synchronize videos with different (but known) frame rates very well. In practice, frame rate information is usually available from video metadata, including for internet videos. Our main limitation is with videos that have varying motion speeds within a single clip (e.g., segments with both slow-motion and fast-motion or non-uniform motion), not with differing constant frame rates across videos.

To demonstrate robustness and adaptability of our method to varying frame rates, we further evaluated our method on the CMU Panoptic dataset underlying 2 challenging scenarios: 1) known varying frame rates and 2) unknown varying frame rates. Videos are sampled at frame rates of 5-30 fps. We directly apply our method without any pipeline changes in scenario 1). For scenario 2), we remove the assumption of known frame rates and follow the extension detailed in the Supp. E. We present experimental results along with synchronization performance on known fixed 30-fps reported in Tab 1 for reference. Our method performs comparably under varying known FPS (median: 51.5 ms) to the fixed 30 fps setting (median: 41.5 ms). When frame rates are unknown, performance degrades gracefully (median: 80.5 ms) due to additional uncertainty, demonstrating robustness of our framework even without frame rate metadata.

---

c) Runtime comparison

Thank you for the suggestion. We have included a runtime and resource efficiency comparison against baselines such as MAST3R, Uni4D, and Sync-NeRF on the Panoptic dataset (15 videos, ~200 frames each) using a single A6000 GPU.

Our method is training-free, and most of the overhead comes from visual cue extraction. While we acknowledge that further optimizations (e.g., lightweight modules) can improve runtime, efficiency is not the primary focus of our work, which targets accuracy and robustness under challenging conditions. With additional computing resources and parallel video preprocessing, the overall running time can be significantly reduced. We view efficiency-focused improvements as an important direction for future work.

d) PoseSync and audio synchronization

Thank you for the suggestion. While both PoseSync [59] and audio-based synchronization [46] are valuable efforts, they are not directly comparable due to differences in assumptions and problem settings.

PoseSync assumes a single moving human and leverages pose priors, whereas our datasets feature multiple moving people or non-human subjects (e.g., pigeons in 3DPOP and dynamic objects in sync-NeRF), making direct comparison infeasible. That said, we agree it would be useful to include PoseSync on single-human datasets, and will add such experiments in the final version (beyond the rebuttal timeline).

Audio-based methods rely on additional audio input and make different assumptions. While orthogonal to our visual-only setup, we agree it is important to clarify this distinction, and we will add a more detailed discussion in the Related Work section to better position our assumption-free, general-purpose approach.

---

e) Additional prior works

We respectfully note that our selected baselines are competitive and representative for general video synchronization—an underexplored problem—spanning both traditional geometric and deep learning approaches. While we acknowledge the reviewer’s concern, the comment (“shall be many” and “seem not enough”) is a bit general to us. We would greatly appreciate any concrete recommendations on additional methods relevant and feasible for evaluation in our setting of generic multi-view time synchronization, and we will gladly consider incorporating them in our revision.

In the meantime, we will expand the Related Work section to better articulate our baseline choices and further situate our contributions within the broader literature.

a) Sensitivity to input quality

We clarify that AUC@100 in Tab. 3 is an intermediate metric across all video pairs, including those with opposite viewpoints or no temporal overlap. For ground-truth evaluation (1st row in Tab. 3), we use dataset-provided GT poses and correspondences for all pairs—regardless of overlap—to demonstrate that our system produces near-perfect output when given perfect input.

In practical scenarios, the more relevant comparison is between the bottom row and 2nd rows. On the challenging EgoHumans dataset, our framework achieves a median error of 46.6 ms, which is close to the 28.6 ms oracle result obtained with ground-truth poses—demonstrating robust performance with imperfect inputs.

As further shown in 5th row in Tab. 4, our method leverages the energy landscape to filter spurious pairs, highlighting its robustness to imperfect inputs.

---

b) Quantatitive performance

The mean synchronization error ( ~100 ms) is heavily influenced by a small number of extremely challenging camera views, as we did not exclude any views from our evaluation. The Supplementary Video provides visual examples to highlight the difficulty of synchronization (e.g. synchronization between third-person and egocentric cameras with rapid motion blur). Notably, the median error is much lower (~40 ms), indicating that our method achieves strong synchronization accuracy for the majority of cameras. We also report error percentiles on EgoHumans dataset below, showing that the higher mean is driven by a minority of extreme cases.

---

c) Diverse frame rate handling

First, we would like to clarify that our method can synchronize videos with different (but known) frame rates very well. In practice, frame rate information is usually available from video metadata, including for internet videos. Our main limitation is with videos that have varying motion speeds within a single clip (e.g., segments with both slow-motion and fast-motion or non-uniform motion), not with differing constant frame rates across videos.

To demonstrate robustness and adaptability of our method to varying frame rates, we further evaluated our method on the CMU Panoptic dataset underlying 2 challenging scenarios: 1) known varying frame rates and 2) unknown varying frame rates. Videos are sampled at frame rates of 5-30 fps. We directly apply our method without any pipeline changes in scenario 1). For scenario 2), we remove the assumption of known frame rates and follow the extension detailed in the Supp. E. We present experimental results along with synchronization performance on known fixed 30-fps reported in Tab 1 for reference. Our method performs comparably under varying known FPS (median: 51.5 ms) to the fixed 30 fps setting (median: 41.5 ms). When frame rates are unknown, performance degrades gracefully (median: 80.5 ms) due to additional uncertainty, demonstrating robustness of our framework even without frame rate metadata.

---

d) Optimization on low frame rates

While our intermediate pairwise offset estimation operates at discrete timestamps, the final synchronization output is continuous in time. This design reflects the frame-based nature of input videos, point tracks, and camera pose estimates.

We also experimented with interpolating point tracks and camera poses to enable continuous-time matching. However, we observed no significant performance improvement, likely because the discrete matching better couples with the observed data.

To test robustness to low frame rates, we downsampled the CMU Panoptic dataset from original 30fps to 15 fps. As expected, performance slightly degrades due to reduced temporal overlap, but our method still achieves reasonable synchronization accuracy, demonstrating resilience under challenging conditions. We want to further add that our preprocessing methods, Co-Tracker and DEVA, are designed for video input and typically assume a high frame rate, making 15 FPS uncharacteristically low for optimal performance.

---

e) Stationary vs. moving cameras

It is not necessarily true that static cameras make better anchors. The major benefit is static cameras often offer more accurate pose estimates, but they may suffer from limited viewpoint or temporal overlap with others. In contrast, dynamic cameras can provide richer visual connections, which our method effectively exploits.

As shown in our supplementary videos, our approach works well even when all cameras are moving or when pose estimates are noisy, including challenging in-the-wild cases.

To validate this, we conduct an ablation using only dynamic cameras in selected EgoHumans sequences—specifically those without opposing viewpoints (e.g., volleyball, badminton) or with two opposing players (e.g., tennis, fencing). As shown in the below table, the fist column (static & dynamic) leverages both static and dynamics cameras for synchronization while the second column (dynamic-only) only use dynamic cameras. We report the dynamic camera synchronization performance in the table. As show below, the synchronization performance remains strong with only dynamic cameras input, confirming that our method is effective for both static and dynamic camera setups.

We also apply our method to in-the-wild videos (Supp. Fig. 3 and Video), including NBA basketball and FIFA soccer sequences where all cameras are moving. Our method still achieves good synchronization accuracy in these challenging real-world scenarios.

---

f) Use of K-Planes vs. Dynamic 3D Gaussian Splatting

Our method is agnostic to the underlying radiance field representation and can interface with a wide range of dynamic scene reconstruction models, including K-Planes, Dynamic Gaussian Splatting, or others.

We chose K-Planes in our experiments for consistency with Sync-NeRF [24]. We agree that more recent methods such as dynamic 3DGS offer compelling benefits in speed and quality. Incorporating such representations to demonstrate downstream application is a valuable and very possible direction, and we will do this in our revision to further show the versatility of our synchronized outputs for downstream 3D reconstruction and rendering.

---

g) Minor

We will fix the writing issue in revision.

a) Dependency on accurate camera poses

While our method requires predicted camera poses as input, it does not rely on accurate poses for every frame. In practice, a subset of frames with reliable pose estimates is sufficient, thanks to our design—particularly the energy landscape optimization and robust filtering of spurious pairs. This allows the method to remain resilient to pose noise and gracefully degrade under imperfect predictions.

This is empirically supported in two ways:

1. Pose error vs. sync accuracy (table below): Even with large rotation and translation errors (relative angular error in rotation and translation following VGGT) in camera pose estimates, our method maintains low synchronization errors (e.g., 9.6 ms for Fencing and 19.4 ms for Tagging), indicating that pose noise does not directly translate to sync failure.

2. GT vs. estimated poses: As shown in Tab. 3 (2nd and bottom row), using estimated poses on the challenging EgoHumans dataset still yields a median error of 46.6 ms, close to the 28.6 ms oracle result with ground-truth poses—demonstrating robust performance even under imperfect pose inputs.

Further, Tab. 4 (5th row) shows how our method filters out unreliable matches via the energy landscape, enhancing stability.

We acknowledge this limitation in our paper, but clarify that it is not a binary failure mode. We will make this point more explicit in the revision.

Note: High sync error in Tennis is due to distant camera placement, resulting in minimal observable motion (see supplementary video).

---

b) Rapid motion or large viewpoint changes:

We demonstrate real-world effectiveness by applying our method to challenging in-the-wild videos, as shown in the supplementary material (Supp Fig. 3 and Video). These include NBA basketball sequences and FIFA soccer sequences with rapid shooting motion, diverse backgrounds, and varying camera viewpoints.

---

c) Frame rate variations:

First, we would like to clarify that our method can synchronize videos with different (but known) frame rates very well. In practice, frame rate information is usually available from video metadata, including for internet videos. Our main limitation is with videos that have varying motion speeds within a single clip (e.g., segments with both slow-motion and fast-motion or non-uniform motion), not with differing constant frame rates across videos.

To demonstrate robustness and adaptability of our method to varying frame rates, we further evaluated our method on the CMU Panoptic dataset underlying 2 challenging scenarios: 1) known varying frame rates and 2) unknown varying frame rates. Videos are sampled at frame rates of 5-30 fps. We directly apply our method without any pipeline changes in scenario 1). For scenario 2), we remove the assumption of known frame rates and follow the extension detailed in the Supp. E. We present experimental results along with synchronization performance on known fixed 30-fps reported in Tab 1 for reference. Our method performs comparably under varying known FPS (median: 51.5 ms) to the fixed 30 fps setting (median: 41.5 ms). When frame rates are unknown, performance degrades gracefully (median: 80.5 ms) due to additional uncertainty, demonstrating robustness of our framework even without frame rate metadata.

---

d) Venue suitability:

We believe our work is well-suited for NeurIPS, as it addresses a fundamental problem at the intersection of computer vision and machine learning, using tools from visual foundation models and optimization-based inference. Our approach combines insights from geometry, learning, and large-scale visual understanding, with clear implications for downstream applications such as 3D reconstruction and video understanding.