PHI-S: Distribution Balancing for Agglomerative Models

Thank you for your review. We agree that the effect size is small on a per-benchmark basis, but rather argue that the performance across a broad spectrum of tasks does meaningfully separate different normalization techniques. Because we’re not introducing any new parameters or data, and because the student model is extremely compressed, it’s unsurprising that the effect sizes can be small, but that doesn’t make them irrelevant. Where PHI-S truly shines is in its ability to match all teachers simultaneously (figure 3, tables 3, 11, and 15) without introducing additional parameters (based on the argument that the linear projection of PHI-S can be rolled into the final adaptor linear layer, thus being representationally equivalent). We also demonstrate some interesting properties of the errors of the student model in table 5, showing how the errors are more uniformly balanced, which can be a useful property when applying euclidean measures (e.g. L2 distance) on the produced features. We also show in figure 9 a stability argument, where PHI-S clearly has the most tightly controlled error distributions, which can meaningfully impact learning dynamics.

One thing that seems inarguable from the study is that normalizing the teachers is incredibly important. Once you accept that and introduce the code to support this, then there’s not really much difference in algorithmic complexity between regular standardization and PHI-S, but PHI-S makes the student model better.

Thank you for your detailed review, particularly catching the discrepancy in your second question. At a high level, we feel as though PHI-S is applicable to any situation where you’re learning one or more fixed multivariate distributions. AM-RADIO (or the other agglomerative approaches) fall into this category, and we chose to go with AM-RADIO because it’s been around the longest. Learning a fixed distribution could also be applied in settings such as quantization, and we believe PHI-S has a useful grounding here as it would minimize quantization errors across dimensions since the distribution is balanced across. This is actually where the previous literature on Hadamard matrices in our related work section focuses, although they don’t seem to combine it with PCA, which means they aren’t identically spreading the variance. Getting an apples-to-apples comparison across the agglomerative papers is hard because the space hasn’t standardized on a single setting in the same way that DataComp did for CLIP models. Instead, we wanted our study to be internally consistent, which is Table 1, and then we wanted to demonstrate the relevance in Table 2 by showing that we can ultimately get a very strong ViT-L/16 model. We agree that having an apple-to-apples with at least AM-RADIO would be nice, however, their ViT-H is quite costly to train, and it was prohibitive for the scope of this work. In the following we go through the details of your feedback:

Contribution Weakness: While we agree that Theia also studied standardization, PHI-S has applications beyond just what we studied in this paper. It could similarly be applied anywhere you’re trying to learn a fixed distribution, which we’re seeing similar techniques recently with quantization [10,11, 12], albeit they rely on random matrices as an approximation to PCA. Our paper provides an exact solution which is suitable in situations where the distribution is not changing. We also argue that while Theia applies standardization, they don’t deeply analyze the choice, and this study serves to reinforce the application from both a theory and applications standpoint.

Experimental Weakness: While we do include table 2 as a means for grounding our results with the general literature, it wasn’t our goal to exactly replicate AM-RADIO, but rather rely on the fundamentals introduced in table 1 as a means for comparing the different techniques in isolation of a bunch of design choices. AM-RADIO’s ViT-H model is extremely expensive to train, and thus attempting to exactly match it was beyond our computational budget at the time, which is why we made choices such as multi-stage training for table 2 simply to reduce the computational burden. The net result is that we get rather strong results, which table 2 showcases, but really rely on table 1 as the crux of the study where fair comparisons are made.

Question 1: Because PHI-S is a rotation and uniform scale, one way to think about what it’s doing is implicitly applying loss weights to the different teacher’s features, and a result of this re-weighting is that the summary losses are impacted. Roughly, this would be similar to loss weighting these components by $\alpha^2$ from table 8. Similar to our answer below about SAM metrics, it’s not necessarily surprising that changing the weighting between losses will affect benchmarks that are quite sensitive to particular models, in this case, the summary loss for DFN CLIP/SigLIP. The argument we’re trying to make with this paper is that it’s hard to even address high-level balancing across teachers without first getting them onto a level playing field, and we’re arguing that PHI-S does the best job across those methods studied of balancing their distributions.

Question 2: These differences were a result of transcription errors. We’ve corrected both tables, and greatly appreciate your catching of the mistake.

Question 3: This has to do with how dominant SAM’s feature distribution is relative to the other models. In effect, when applying simple MSE, most of the loss weight is applied to SAM, which you can see in figure 3. So the baseline MSE model is learning SAM much better, at the expense of the other models. Tables 13 and 16 show that basically any type of balancing (as they’ll all reduce the weight of SAM) has a negative effect on the SAM COCO benchmark. Because this benchmark relies on exactly (or at least acceptably) approximating the real SAM model, it’s expected that having a higher SAM MSE (figure 3) would lead to reduced SAM COCO metrics.

References:

[10] Ashkboos, Saleh et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.”

[11] Tseng, Albert et al. “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks.” ArXiv abs/2402.04396

[12] Shah, Jay et al. “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision.” ArXiv abs/2407.08608 (2024): n. pag.

We have also updated the paper in section A.10 and associated Table 21, which is an exact reproduction of AM-RADIO's training recipe, aside from using the smaller ViT-B and ViT-L models, and also applying PHI-S+MSE loss. This allows for a direct comparison to their results. Zero shot and dense perception metrics improve slightly over our SigLIP model with this change, while VLM metrics degrade. However, the ViT-L model is still generally better than AM-RADIO's ViT-H at half the parameter count. This demonstrates that the positive changes in quality are not attributable to the use of SigLIP, but rather the specific topic studied in the paper, which is distribution normalization across the teachers. Training a ViT-H was still outside of our budget for this study, however we believe that the trend in scalability was clear in both AM-RADIO and our study, and so a ViT-H would almost certainly improve further upon our produced ViT-L.

First, I would like to thank the authors for the rebuttal and responding to my questions. After reading the rebuttal and other reviews, my main concern regarding the submission still remains. Regarding the scope of the work, I want to acknowledge the author's response. The proposed PHI-S normalization may or may not be applicable to other domains such as quantization. However, that would need extensive experimental results to determine the effectiveness (specifically for quantization that Hadamard based approaches already exist such as [10]) which will improve the quality of the work but requires major changes to the paper which is beyond the scope of this submission. As of this version, the submission introduces PHI-S as a normalization technique to improve multi-teacher distillation and specifically the AM-RADIO framework and provides experiments only for this setup. Given this, I expected stronger experimental results through head to head comparisons with AM-RADIO. Currently there are mixed results compared to the baseline on downstream tasks and multiple convoluted factors in experiments in various tables throughout the paper that makes it hard to have a fair understanding of the effectiveness of the method for this specific task. I also share some of the concerns raised by reviewer QTud on this front (i.e. the use of different teachers throughout the paper and the new results in Table 21 that shows the impact on downstream tasks, besides the use of different models between the proposed method and the baseline). Consequently, I would like to keep my rating.

Thank you for your in-depth review. We agree that the correlation between fidelity and downstream benchmarks is not 1-to-1, which can be seen in how the relative differences on particular benchmarks is not straightforward. We opted to rely on summarizing the entire suite of benchmarks, under the assumption that a model that tends to perform better across the suite would tend to be a more generalizable model. Rank works well in this regard because we don’t have to normalize the benchmark units in order to aggregate a different way. The primary reason we have the “Avg No COCO” and “Avg No MSE/COCO” columns in Table 1 is so that we could demonstrate that PHI-S works without relying on outlier scores in the MSE task. However, we didn’t have a column which includes everything except for MSE, which we’ve rectified in the updated draft. The added column “Avg No MSE” in table 1 has a consistent conclusion as before, which is that PHI-S does the best on average, followed by global standardization. We did the same thing for table 4, which still has PHI-S as the best method, and in this case, regular standardization remains in second place. We have a detailed response below.

Weakness 1: We’re applying distribution balancing to the spatial features of the teacher models, whereas zero shot accuracy is affected by the summary loss (always cosine) between the student and teacher. The CLIP teachers have a supervised summary feature, and it’s trained using cosine similarity with their text encoders. DINOv2 also uses cosine similarity for its summary tokens. Presumably this is why AM-RADIO chose to use cosine similarity loss for the summary of all teachers (SAM withstanding). There is a tension between the summary losses (cosine) and the feature losses (studied in our paper), where changing the weight of one will impact the others.

Weakness 2: There are two different statements that the paper is making. The central table for the purposes of the paper is actually table 1, where the setup is tightly controlled between different feature distillation losses, and thus it’s a fair comparison. Table 2 is instead showing that we were able to apply the techniques studied to produce an agglomerative model that is competitive with AM-RADIO, but at 15% and 49% of the parameters for ViT-B and ViT-L respectively. It’s essentially a hook for “why should I care” as opposed to the central result of the study.

Weakness 3: We used the comparative ranks as it’s not clear which benchmarks are most important, and also how their magnitude changes relative to each other is important. Instead, the average rank argument is that across a large swath of benchmarks, a better model will tend to do better than its peers across a wider range of tasks. A major problem with agglomerative modeling (or even foundation models in general) is that the downstream use case is unclear, and so the choice of which metrics to target is subjective. If we substantially changed the weights across all summary and feature losses in favor of say, DFN CLIP’s summary loss, then we’d expect to get very strong zero shot (and probably kNN) accuracy, but it would come at the expense of the dense tasks. Applying AdaLoss to the MSE baseline actually does have an effect similar to the standardization methods, as it’s applying the inverse expected loss. This is why we also studied AdaLoss + PHI-S, and found improved teacher matching metrics (table 11) across the board, suggesting that its inclusion, all things equal, yields students with higher fidelity. All said, however, AdaLoss tended to work worse across the full task suite as compared to the standardization methods (Global, Standardize, PHI-S) aside from classification where it strongly produces the best results (see for example the Probe 3D metrics in table 13).

Weakness 4: We disagree on the claims of limited novelty. As for the comparison against AM-RADIO, the entire study focuses around a single design decision in their paper, their equation 3. So we’re leveraging their work, not necessarily comparing against it, aside from in table 2 where we eventually bring the details full circle. While the applications of Hadamard matrices are not new, to the best of our knowledge, combining the eigenvector projection of PCA with the Hadamard rotation as a method to get identical variance across the dimensions of a multivariate distribution is entirely novel. Even Hadamard whitening (as described in [3], but also [5]) is not the same as we’re employing it here. In [3, 5] they’re spreading the error in a similar motivation as ours, but not applying statistical whitening (multiplying by the inverse root eigenvalues). They’re especially not using the Hadamard matrix for statistical standardization, nor are they motivated by the isotropic property of our approach.

Weakness 5: We agree that DINOv2 is a hugely valuable model. Studying the benefits of the teacher models, or even teacher selection itself, is out of scope for this study. There is some recent evidence [6] that suggests that AM-RADIO enjoys better generalization and transfer learning than DINOv2. Theia [7] does study teacher inclusion, however it’s also a relatively small scale study, and they’re not trying to build a general foundation model, but a limited robotic one as evidenced by their exclusion of the summary tokens.

Weakness 6,7: Thank you for the formatting feedback. We will incorporate it into the draft.

Question 1: We explored combining AdaLoss with PHI-S (tables 11-14), and found that teacher MSE is reduced further when additionally applying PHI-S. We agree that AdaLoss marks a large improvement over the baseline MSE without balancing, but disagree that it’s generally competitive against naive weighting combined with standardization.

Question 2: It’s actually the projection head of DINOv2 that is L2 normalized, but the backbone features themselves are unconstrained, and it’s the latter that we’re trying to match. SAM has supervised features at the output, but we’re feature matching before the neck, where they’re also unconstrained. DFN CLIP and SigLIP don’t apply any supervisory signal to the features. We think it would be easy to construct an example where it does make sense to take the final operator of the teacher into account before matching. This is exactly why the summary loss is cosine across the board, because the teacher models used cosine for the summary tokens, and thus it’s possible (or even likely) that the vector magnitudes are meaningless. It seems entirely plausible that LayerNorm could benefit from a boutique normalization technique that operates better for it. What we found in figure 2 was that all of the teachers had mostly gaussian distributions with various means and variances across the dimensions, which is something PHI-S is well designed to handle.

Question 3: This is currently the major black box of agglomerative models. If the adaptor head of the student perfectly matched a teacher, then the student would have the same behaviors as the teacher, but only when still applying that adaptor. Once you introduce a second teacher, even assuming that it also perfectly matched that teacher, it’s unclear what properties the student backbone would exhibit, assuming that it wasn’t wide enough to represent both teachers exactly. In this case of our study, our ViT-B only has a width of 768, and ViT-L a width of 1024. DINOv2-g alone has a width of 1536, meaning that our student (in the case of ViT-B) is only half the width of just one of its teachers. What we do consider to be important, all things equal, is that higher teacher fidelity (measured here as lower MSE) is preferable. Our study doesn’t introduce a change of data, and we constrain the experimental setting to be identical everywhere except for the loss function (and associated weights), which makes it fair to care about fidelity a lot (tables 3, 11, and 15). We have also added section A.9.4 with associated tables 18 and 19 to attempt to unify fidelity across the teachers in a way in which it can be aggregated. We agree that finding teachers that are distinctly good at certain tasks and correlating them with their losses is a good topic of future work, but teacher selection was largely outside the scope of this study.

References:

[5] Anthony Trioux, François-Xavier Coudoux, Patrick Corlay, M Gharbi. A comparative preprocessing study for softcast video transmission. 2018 9th International Symposium on Signal, Image, Video and Communications (ISIVC), Nov 2018, Rabat, Morocco. pp.54-59, ff10.1109/ISIVC.2018.8709171ff. ffhal-03335982f

[6] Drozdova, Mariia et al. “Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition.” ArXiv abs/2410.02069 (2024)

[7] Shang, Jinghuan et al. “Theia: Distilling Diverse Vision Foundation Models for Robot Learning.” ArXiv abs/2407.20179 (2024)

In response to weakness (2), we have also updated the paper in section A.10 and associated Table 21, which is an exact reproduction of AM-RADIO's training recipe, aside from using the smaller ViT-B and ViT-L models, and also applying PHI-S+MSE loss. This allows for a direct comparison to their results. Zero shot and dense perception metrics improve slightly over our SigLIP model with this change, while VLM metrics degrade. However, the ViT-L model is still generally better than AM-RADIO's ViT-H at half the parameter count. This demonstrates that the positive changes in quality are not attributable to the change in recipe (e.g. SigLIP or stages), but rather the specific topic studied in the paper, which is distribution normalization across the teachers.

Thank you for your review. At a high level, we relied on AM-RADIO to motivate the agglomerative model paradigm in general, which allowed us to focus solely on improving it. We think that all of the issues you raised in the weaknesses section are indeed quite valid questions regarding agglomerative modeling, and we think that your first two identified weaknesses are worthy of much deeper and dedicated studies. There is emerging evidence that these models do generalize well in practice, which we talk about in our specific answer regarding your first identified weakness.

Weakness 1: The AM-RADIO paper does most of the motivating for distilling multiple foundation models, and this work largely assumes that the technique is important based on AM-RADIO’s assertion. What they found was that different VFMs have different strengths and weaknesses, owing largely to how they were trained. The CLIP family of models is great for holistic reasoning, and seems to have a strong sense of semantic alignment with language, however they struggle significantly with dense perception tasks (e.g. semantic segmentation, or the 3D vision tasks tested here). DINOv2 is pretty much the gold standard for perception right now, but it struggles a lot with OCR (e.g. TextVQA in Table 1 of AM-RADIO). It indeed seems to be the case that fusing multiple teachers into these smaller students works well across a spectrum of tasks [6, 8, 9], and in all of these instances, the student model was smaller than even just DINOv2, ignoring the other teachers.

Weakness 2: We agree that choice of dataset can potentially be significant. We purposefully chose to keep the dataset constant in this paper so that we could solely focus on the topic of distribution normalization. Teacher selection and dataset selection are topics that need larger treatments that couldn’t fit into the scope of this study.

Weakness 3: Are you referring to how quickly our estimates of the rotation matrix and alpha scalar converge? We’re currently just computing these estimates over the first 3 million examples seen during training (first 3k mini-batches), but agree that it might be interesting to see how quickly those values converge. We did run into issues with PyTorch’s eigh call not returning stable eigenvectors, which was a problem for all methods relying on that function, except for ZCA, as the inverse rotation was counteracting the instability.

Weakness 4: We agree that ultimately increasing the scale of the model would be good, but we were bound by computation constraints, as you identified in the next weakness.

Weakness 5: Finding PHI-S is mostly negligible, as we only "learn" it over the first 3k steps (of 300k), and afterward the application of it is a GEMM at each iteration, which is very small relative to the actual model. During inference, all of the studied normalizations can be rolled into the weights of the teacher adaptors, making them free. That said, explicitly, running the GEMM on an A100 with an input size of (32, 729, 1280) (from DFN CLIP) and PHI-S transform (1280, 1280) takes about 0.46ms, and so we’re looking at roughly 2ms/step of overhead across the four teachers, compared against the approximately 355ms/step of the general training harness. We also note that while training the model is expensive, and while specifically trying to make the training cheaper wasn’t a focus of the study, we did improve over AM-RADIO’s recipe which is 17,408 GPU-hours.

References:

[6] Drozdova, Mariia et al. “Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition.” ArXiv abs/2410.02069 (2024)

[8] Lu, Yuxiang et al. “Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning.” (2024).

[9] Guo, Pinxue et al. “VideoSAM: Open-World Video Segmentation.” (2024).