From Pretraining to Pathology: How Noise Leads to Catastrophic Inheritance in Medical Models

Reviewer PCzQ

We thank the reviewer PCzQ for your constructive feedback. Below we address each point in turn.

Q1: How the target statistics (τₛ and τₖ) are chosen and how sensitive the method is to these values, noting that the explanation of "obtained from a clean model" or "set empirically" is unclear and could affect practical deployment.

A1: We sincerely thank the reviewer for raising this point. To better clarify, we conducted additional sensitivity analyses on τₛ and τₖ.

The results indicate that SKD remains stable across a wide range of τₛ and τₖ values. This observation aligns with our design intuition: the effectiveness of SKD mainly comes from counteracting representation collapse rather than relying on precise target values. Although we initially set τₛ and τₖ to the negative values of those measured from a clean model, we found that other reasonable settings (as long as they are not extreme) yield very similar performance. In practice, these values can indeed be set based on the skewness and kurtosis estimated from a clean model, but the method is not sensitive to the exact values—any setting that helps suppress representation collapse works well. Therefore, for simplicity and consistency, we fixed τₛ = 0 and τₖ = 0 in all main experiments.

Q2: The paper lacks comparisons with other methods for dealing with noisy labels. Although it includes results with LP and NML, this comparison alone may not provide a comprehensive assessment of the method's robustness.

A2: We appreciate the reviewer for raising this point. We would like to note that catastrophic inheritance in noisy medical pretraining is still a new research problem, and there are very few existing approaches tailored to this issue—especially for frozen foundation models.

To address this gap and provide a more thorough comparison, we have expanded our baseline set to include GCE (Generalized Cross-Entropy)[1], a widely used method from the noisy label learning field. Although GCE was not specifically developed for catastrophic inheritance, it serves as a strong benchmark for evaluating robustness. As illustrated below, SKD consistently surpasses all other methods:

Q3: Whether the performance gains are consistent across all classes or mainly driven by certain classes, noting that per-class performance metrics are not provided.

A3: We thank the reviewer for this valuable suggestion. Following your advice, to evaluate whether the observed improvements are uniformly distributed across classes, we analyzed per-class accuracy on the Camelyon17 dataset under 10% label noise.

As shown in the table below, SKD outperforms LP, GCE, and NML across all classes, with no dominant trade-off on minority or majority categories. These results suggest that the performance gains are broadly distributed and that SKD achieves consistent improvements across all class types.

Q4: The method is evaluated on ResNet-50 in Table 1. Have the authors tested it on other CNN or ViT architectures? Providing results on more diverse backbones would support the generalizability claims.

A4: Thank you for this important suggestion. To further support the generalizability of our approach, we have conducted additional experiments on diverse backbones, including large-scale foundation models.

• For vision tasks, we evaluated SKD on ViT-L, a Transformer-based vision backbone that is substantially larger.

• For natural language processing, we extended our experiments to GPT-2, a large Transformer-based language backbone, and tested it on five representative biomedical NER datasets.

The results, summarized below, show that SKD consistently outperforms all baselines on both vision and NLP models, with approximately 2% gains on ViT-L tasks. This further confirms the scalability and generality of our method across diverse architectures and domains.

These findings demonstrate that SKD generalizes well across fundamentally different backbones and task types, including challenging large-scale foundation models.

Q4: In Table 1, it is unclear whether the "YFCC15M" and "ResNet50" refer to datasets or architectures. Could the authors clarify this distinction and make the table more explicit regarding which architecture is pretrained on which dataset?

A4: We thank the reviewer for pointing out this ambiguity. In Table 1, "ResNet-50" refers to the backbone architecture, and "YFCC15M" refers to the dataset used for pretraining.

We agree that the current formatting may lead to confusion. In the final version, we will revise the table caption and column labels to explicitly distinguish between the model architecture and the pretraining dataset.

Thank you again for the helpful suggestion to improve clarity.

[1] Zhang, Zhilu, and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems 31 (2018).

Reviewer Xipk

We thank the reviewer Xipk for your constructive feedback. Below we address each point in turn.

Q1: Whether the observed phenomenon and the effectiveness of the proposed mitigation are specific to the medical domain or also apply to non-medical settings.

A1: We thank the reviewer for raising this important question. Following your advice, we conducted new experiments on three general-purpose vision benchmarks:DTD, CIFAR-10, Office-31.

We first analyzed skewness and kurtosis of feature representations. The results confirm that both statistics decrease as noise increases, which supports our claim that feature distributional collapse is a general phenomenon:

We then compared the downstream classification performance of various methods, including: LP (linear probing), GCE (Generalized Cross Entropy)[1], NML (spectral norm preserving baseline), SKD (ours):

Across all datasets, SKD consistently outperforms alternatives, demonstrating its effectiveness across both medical and general-purpose domains. On CIFAR-10, SKD under 30% noise even surpasses LP trained with only 5% noise, showcasing its ability to counteract heavy label noise.

Q2: The reviewer questions the generalizability of the framework, noting it only addresses random noise and one baseline, and argues that the observed correlation between skewness/kurtosis and performance drop may not imply causation, especially given the weak link to noise ratio in Figure 3.

A2: We sincerely thank the reviewer for this comprehensive and thoughtful comment. Your question touches on several important aspects, and we will address each sub-question one by one in detail:

1). Noise type

Thank you very much for this important suggestion. Following your advice, we have conducted additional experiments to explicitly evaluate robustness under structured noise settings.

We used the ImageNet dataset as a base and synthetically introduced taxonomy-aware hierarchical label noise, where a controlled percentage of image labels were randomly replaced by their WordNet hypernyms. This setting simulates realistic structured mislabeling commonly observed in real-world annotation pipelines. Below is the results:

As shown above, SKD consistently improves robustness across all noise levels, validating its effectiveness in structured label corruption scenarios.

We would also like to emphasize that our main experiments—particularly those on PLIP—already involve realistic, naturally occurring noise. The noisy image–text pairs in PLIP capture uncontrolled web-scale noise, including co-occurrence bias and semantic drift, making it a strong real-world testbed for validating our method.

2). Baseline comparison
Thank you for pointing this out. We would like to clarify that catastrophic inheritance in noisy medical pretraining is a relatively novel research direction, with very few existing methods specifically designed to address this phenomenon.

Given this gap, and to ensure a more comprehensive comparison, we have supplemented our original baselines by including GCE (Generalized Cross-Entropy)[1]—a well-established method from the noisy label learning literature. Although not originally designed for catastrophic inheritance, GCE offers a strong baseline for robustness evaluation. As shown below, SKD consistently outperforms all other methods:

3). Correlation vs. causation regarding skewness/kurtosis

We thank the reviewer for this important comment. To further investigate this connection, we conducted a class-wise analysis of feature distribution shape and intra-class variance (as a proxy for pathology heterogeneity) on DigestPath:

We observe that classes with more histological complexity—such as malignant tumors—naturally exhibit higher skewness and kurtosis, suggesting that these metrics reflect lesion-level heterogeneity, including irregular boundaries and intra-class diversity.

To complement this analysis, we performed a qualitative Grad-CAM comparison between models trained with clean vs. noisy pretraining. We found that activation maps from clean models better captured relevant lesion morphology. This supports the interpretation that the collapse of distributional shape also signals a loss of clinically meaningful structure.

We validated this interpretation under hierarchical label noise, where increasing noise levels led to clear declines in skewness and kurtosis, indicating feature collapse.

These results confirm that skewness and kurtosis reliably reflect feature collapse, even under realistic noise. Full visualizations will be included in the final version.

Q3: The figure 3 is unreadable because of the legend. There is no way to distringuish the mean and the variation because the legends does not show dashed lines

A3: We thank the reviewer for pointing this out. We acknowledge that Figure 3 currently lacks a clear legend, affecting readability. In the revision, we will add an explicit legend to clearly distinguish mean and variation.

Q4: Why skewness and kurtosis are kept if they are unstable in Line 220, and why the ablation shows disagreement loss has limited impact compared to kurtosis loss.

A4: Thank you for highlighting this important point. We apologize for the potential source of confusion and would like to clarify the intended meaning.

In Line 220, the term unstable refers specifically to computing skewness and kurtosis in the logit space, not the feature space used in our method. This distinction is crucial:

• In many medical classification tasks, the logit dimension is very small (e.g., 2 classes), which makes high-order moment statistics such as skewness and kurtosis numerically unstable and less informative when applied directly to logits.
• In contrast, when computed over the feature representations, these statistics are stable, well-defined, and strongly correlated with representation quality, forming the foundation of our SKD regularization approach.

We appreciate the opportunity to clarify this point and will revise the relevant sentence in the final version to avoid confusion.

While the disagreement loss (D) may appear to contribute a smaller relative improvement compared to the skewness–kurtosis regularization (SK) alone, it still provides meaningful complementary gains, with over 1% improvement on both HAM10000 and NIH ChestXray, further reinforcing the effectiveness of our full method.

Q5: How do you compute / define the target statistics τₛ and τₖ?

A5: We appreciate the reviewer for raising this question. Following your advice, we performed additional sensitivity studies on (τₛ) and kurtosis (τₖ).

Our experiments show that SKD is highly robust to the choice of these hyperparameters. This aligns with the intuition that the main contribution of SKD lies in alleviating representation collapse by promoting meaningful skewness and kurtosis. Although τₛ and τₖ were initially set to the negative values of those measured from a clean model, we observed that other reasonable values (as long as they are not extreme) lead to very similar improvements. For simplicity and consistency, we set τₛ = 0 and τₖ = 0 across all main experiments.

Q6: The reviewer asks about the behavior of skewness and kurtosis under low noise levels (<10%).

A6: Thank you for the thoughtful comment. As noted earlier, we have added results under 1%, 2%, and 5% hierarchical label noise, including skewness/kurtosis and performance comparisons across LP, GCE, NML, and SKD.

These results show that even mild noise leads to a decline in skewness/kurtosis, indicating early collapse, and that SKD remains effective in this regime. We will revise Figure 3 and clarify these trends in the final version.

[1] Zhang, Zhilu, and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems 31 (2018).

Reviewer rgdp

We thank the reviewer rgdp for your constructive feedback. Below we address each point in turn.

Q1: Do authors get any sense of the importance of catastrophic inheritance on different types of downstream task? e.g. how much more severe is the problem on an MRI segmentation task compared with an X-Ray text-summarization task? Is the problem also imaging domain variant - in more difficult domains such as ultrasound, and histopathology, is the problem exacerbated compared with images of larger structures, e.g. MRI?

A1: We sincerely thank the reviewer for this insightful question. We could not agree more that catastrophic inheritance is likely to manifest differently across task types and imaging modalities, and that this deserves a much deeper analysis.

Our current study primarily focuses on medical image classification tasks (e.g., HAM10000, Camelyon17, NIH ChestX-ray), where we observe a clear flattening of feature distributions—quantified by reduced skewness and kurtosis—under noisy pretraining. We also include medical NER experiments (e.g., PubMedBERT) to show that similar degradation patterns occur in language models, suggesting the phenomenon is not modality-specific.

That said, we strongly believe that medical segmentation tasks (e.g., MRI or ultrasound segmentation) may exhibit distinct and potentially even more severe forms of catastrophic inheritance, because noise propagates differently in dense prediction settings and these modalities often involve inherently higher uncertainty. This is a separate and highly valuable research problem, and we are committed to investigating it in future work with the level of depth it deserves.

We are very grateful to the reviewer for highlighting this direction—it aligns closely with our long-term vision.

Reviewer 7kcL

We thank the reviewer 7kcL for your constructive feedback. Below we address each point in turn.

Q1: The current analysis focuses on synthetic uniform label noise; extending it to structured or semantic noise (e.g., hierarchical or co-occurrence-based) would better reflect real-world medical scenarios.

A1: Thank you for this valuable suggestion. Following your advice, we have extended our analysis to include hierarchical mislabeling to better reflect structured noise. Specifically, we used the ImageNet dataset as a base and synthetically introduced taxonomy-aware label noise: a controlled percentage of image labels were randomly replaced with their WordNet hypernyms, simulating realistic structured mislabeling scenarios common in annotation pipelines (e.g., “Labrador Retriever” → “dog”).

We then trained a ResNet-50 model from scratch under different noise levels and evaluated four methods:LP(standard linear probing), GCE[1](Generalized Cross-Entropy), NML(spectrum-preserving baseline), SKD(our proposed method)

Results show the effectiveness of SKD under structured, taxonomy-aware noise:

Finally, we would like to highlight that, while part of our study uses synthetic uniform noise for controlled experiments, our main evaluations—particularly those on PLIP—already involve real-world, naturally occurring noise. The noisy image–text pairs in PLIP contain uncontrolled web-scale noise, making PLIP a strong real-world testbed for validating our method.

Q2: The current experiments focus on mid-sized models; evaluating on large foundation models is needed to demonstrate scalability and generalizability.

A2: Thank you for this important suggestion. Following your advice, we have conducted additional experiments to verify our method on large-scale foundation models.

• For vision task, we evaluated on the ViT-L backbone.
• For natural language processing, we extended our experiments to GPT-2.

The results demonstrate that SKD continues to achieve consistent improvements, further validating its scalability.

Q3: While impactful in the medical domain, its generalization to non-medical tasks is not explored (but appropriately scoped).

A3: Following your advice, we have conducted additional experiments on three widely-used non-medical vision benchmarks:DTD (Describable Texture Dataset), Office-31, CIFAR-10.

The results consistently show that SKD improves performance under noise across all datasets, further validating our hypothesis. Notably, on CIFAR-10, SKD under 30% noise still outperforms LP under 5% noise, demonstrating its strong ability to mitigate severe label corruption.

Q4: Builds on prior work like NML and SVDEntropy; while differentiated, it's an incremental rather than revolutionary advancement.

A4: Thank you for this thoughtful comment. While our work relates to prior studies such as NML, it introduces a fundamentally new perspective and addresses a critical gap in the literature.

1. New problem setting: We systematically study catastrophic inheritance under noisy medical pretraining, a practically important yet underexplored challenge—especially relevant for large-scale medical vision-language models like PLIP. This setting, where frozen noisy features are transferred to downstream classifiers, is distinct from conventional noisy-label learning and demands targeted solutions. Our framework directly targets this high-stakes, underexplored setting.

2. Novel perspective: Our method introduces the use of skewness and kurtosis as high-order, distribution-level signals to diagnose and mitigate feature collapse. This goes beyond spectrum-preserving or low-order statistics used in prior work, providing a structurally richer and more interpretable signal to guide learning. To our knowledge, this is the first work to explicitly regularize distributional shape in the context of representation collapse.

3. Broad impact and generality: Our method achieves consistent and significant improvements over baselines on a range of challenging medical datasets (e.g., HAM10000, Camelyon17, NIH ChestXray), and scales effectively to large foundation models (e.g., ViT-L, PLIP, GPT-2), and shows consistent gains across both vision and NLP domains. These improvements hold even on large-scale foundation models and challenging medical datasets.

We will emphasize these unique contributions more clearly in the revision to better highlight the problem focus, conceptual novelty, and broad empirical benefits of our work.

Q5: Target Statistics: How sensitive is SKD to the target skewness/kurtosis values? Are these computed from a clean model, and what happens if these targets are misestimated?

A5: Thank you for raising this important question. Following your suggestion, we conducted additional sensitivity experiments for the two hyperparameters on HAM10000 (the target skewness and kurtosis values, denoted as τs and τk).

The results demonstrate that SKD is not sensitive to these hyperparameters, which is expected because the primary effect of our method is to counteract representation collapse by encouraging non-degenerate skewness and kurtosis. While these targets were initially set to the negative values of the skewness and kurtosis computed from a clean model, we found that other reasonable values (not excessively large) achieve similar mitigation.

In all our main experiments, we simply set τs = 0 and τk = 0 for consistency. Below are the sensitivity results:

Q6: Interpretability & Clinical Relevance: Can you connect changes in skew/kurtosis with specific pathology structures (e.g., lesion shape variability)? This could strengthen interpretability.

A6: We thank the reviewer for this thoughtful question. While our core analysis focuses on high-order statistics of feature distributions, we agree that bridging these changes with observable pathology structures would strengthen interpretability.

To investigate this connection, we conducted a class-wise analysis of feature distribution shape and intra-class variance (as a proxy for pathology heterogeneity) on DigestPath. Results are summarized below:

We observe that classes with more histological complexity—like malignant tumors—naturally exhibit higher skewness and kurtosis, suggesting that these metrics may reflect lesion-level heterogeneity such as irregular boundaries or intra-class diversity.

To complement this, we performed a qualitative Grad-CAM analysis comparing models trained under clean and noisy pretraining. We found that activation maps under clean conditions (with higher skew/kurtosis) better captured relevant lesion morphology, while those under noisy pretraining (with collapsed skew/kurt) became more diffuse and less structure-aware. This supports the interpretation that the collapse of distributional shape also signals a loss of clinically meaningful structure.

Due to rebuttal constraints, we omit figures here but will include visualizations in the final version. Thank you for encouraging this line of interpretability exploration.

Q7: Since logit entropy increases with noise, have you considered integrating uncertainty calibration methods to complement SKD?

A7: We thank the reviewer for this valuable suggestion. Following your suggestion, we applied Temperature Scaling (TS)[2], a widely used post-hoc calibration technique that adjusts softmax confidence via a learned temperature parameter without modifying model weights.

The table below summarizes classification accuracy:

We observe that SKD already yields robust predictions across noise levels, and temperature scaling provides a modest but consistent boost. We appreciate the reviewer’s suggestion and will include this analysis in the final version.

[1] Zhang, Zhilu, and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems 31 (2018).

[2] Guo, Chuan, et al. "On calibration of modern neural networks." International conference on machine learning. PMLR, 2017.