Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology

Thank you for your thorough technical assessment and clear articulation of our key contributions. We're particularly pleased that you recognized how our dual-conditioning mechanism addresses the fundamental information loss problem in text/embedding-based methods, and that you highlighted the contribution of processing 11,765 TCGA WSIs without manual annotations. Your acknowledgment of our comprehensive evaluation strategy confirms our commitment to establishing clinical relevance, and we appreciate your recognition of our open science approach.

---

WSI-level structural coherence: Our model maintains excellent structural coherence across large WSIs through several design mechanisms. The dual-conditioning approach specifically addresses this challenge: semantic maps provide global spatial organization while visual crops ensure fine-grained morphological consistency at tissue interfaces. Our heterogeneous sampling strategy targets tissue boundaries and transition zones where coherence is most critical.

Unlike semantic-only approaches that can generate artificial boundaries, our visual crop conditioning preserves authentic tissue transitions observed in real histopathology. The pathologist evaluation specifically assessed structural coherence, with synthetic samples receiving comparable scores to real tissue for histological structural detail, validating WSI-level organization essential for diagnostic applications.

---

UNet vs DiT architectural choice: U-Net outperforms DiT in data efficiency, training simplicity, spatial precision, and computational cost, making it better suited for tasks like medical image segmentation, where labeled data is limited and fine-grained localization is critical. Histopathology presents unique characteristics: (1) limited spatial diversity compared to natural images enables faster UNet convergence, (2) tissue morphology has inherent structural constraints that facilitate learning, (3) diagnostic focus requires fine-grained detail preservation, where UNet's skip connections provide advantages. Our UNet achieved target performance metrics with downstream segmentation reaching within 1-2% of real-data performance.

We're actively exploring DiT integration for future work, expecting improvements in both synthesis quality and training efficiency. The dual-conditioning framework is architecture-agnostic and should transfer seamlessly.

---

Multi-modal adaptation capabilities: Our framework is inherently stain-agnostic by design. The core innovation, using raw visual crops rather than processed embeddings, preserves authentic staining characteristics regardless of protocol. For IHC stains, visual crops maintain antibody-specific colorations and cellular localization patterns. For special stains, crops preserve characteristic color distributions essential for accurate synthesis.

Cross-modal adaptation requires training on corresponding annotated datasets, but the dual-conditioning architecture remains unchanged. The framework could be extended for paired modality synthesis (generating H&E conditioned on IHC patterns or vice versa), enabling cross-stain validation studies and addressing practical clinical challenges like missing stains or batch effects between laboratories.

---

Computational accessibility strategies: While computational requirements are substantial, they're designed to democratize rather than restrict access through strategic resource sharing. The costs are front-loaded investments enabling broad community benefit:

• Optimization Strategies: 8×H100 setup reduces 3-month feature extraction to ~6 days. Cloud deployment makes intensive computations accessible without local infrastructure. Pre-trained models serve as shared resources for synthetic data generation without requiring local training.
• Practical Deployment: Inference requires only standard GPU hardware (1.2s per image on single GPU), enabling real-time generation at most research institutions. The "train once, deploy everywhere" model particularly benefits resource-limited institutions.

---

Rare pattern coverage and representation: Our unsupervised clustering approach naturally accommodates rare morphological patterns through several mechanisms:

• Comprehensive Coverage: Processing 634M patches from 11,765 WSIs ensures rare cases are represented rather than overlooked. Feature-based clustering creates distinct groups for morphologically unique patterns regardless of prevalence.
• Clustering Methodology: While textbooks may define only 4 main tissue types (e.g., epithelial, connective, muscle, nervous), the actual visual and structural variability within and across these categories—due to factors like cell density, staining variation, disease progression, and microenvironment—is immense; thus, using 100 clusters allows us to process a much richer set of discriminative visual patterns, capturing subtle phenotypic differences and context-dependent tissue morphologies that are essential for robust training, especially in tasks like segmentation.
• Validation Evidence: The confusion matrix analysis, in the Supplementary File page 8 Figure 5, shows misclassifications typically occur between morphologically similar clusters rather than complete failures, indicating effective rare pattern representation.

We're grateful for your insightful technical analysis, particularly your recognition that our dual-conditioning approach directly preserves fine-grained pathological details lost in prior embedding-based methods. Your appreciation of our multi-faceted evaluation strategy—from blinded pathologist assessment to eight foundation-model encoders—validates our comprehensive approach to establishing clinical relevance in this critical computational pathology bottleneck.

---

Computational cost limitations: While the computational requirements are substantial, they represent a strategic one-time investment with significant long-term benefits for the computational pathology community. Our analysis shows multiple optimization pathways:

• Hardware Acceleration: Using modern 8×H100 setups reduces the 3-month TCGA feature extraction to approximately 6 days (H100 provides 2-3× speedup over A100). The clustering phase (48 hours on 124-core CPU) scales linearly with additional cores and is accessible through cloud computing.
• Community Resource Model: Once trained on TCGA, our models serve as shared resources enabling institutions to generate synthetic data without local retraining. This "train once, deploy everywhere" paradigm particularly benefits resource-limited institutions that gain access to state-of-the-art synthesis capabilities without prohibitive computational investments.
• Practical ROI: Inference efficiency (1.2s per image on single GPU) enables real-time synthetic data generation. A single trained model can produce thousands of annotated training samples daily, providing substantial return on the initial computational investment.

---

Cross-modal adaptability: Our framework is inherently stain-agnostic by design. The core innovation, using raw visual crops rather than processed embeddings, preserves authentic characteristics regardless of staining protocol. For IHC stains, visual crops maintain antibody-specific brown/blue colorations and cellular localization patterns. For special stains like Masson's trichrome, crops preserve characteristic blue collagen and red muscle fiber appearances essential for accurate synthesis.

The dual-conditioning architecture requires no modification for cross-modal adaptation, only training on corresponding annotated datasets. We're planning validation studies on IHC datasets and special stain collections to demonstrate this versatility. The framework could even enable paired modality synthesis (generating H&E conditioned on IHC patterns or vice versa).

---

Clustering quality analysis: We've implemented comprehensive clustering validation to address quality concerns:

• Quantitative Metrics: Silhouette analysis shows average score of 0.41 across 100 clusters indicating well-separated, cohesive groupings. Our tissue classifier achieves 93.7% accuracy on held-out test set, demonstrating morphological coherence within discovered clusters.
• Cluster Number Rationale: In pathology, tissue structures exhibit highly heterogeneous and hierarchical patterns, so using a large number of clusters (e.g., 100) allows the model to capture subtle morphological variations, rare phenotypes, and fine-grained tissue distinctions that would be lost with a smaller, overly coarse clustering. 100 clusters balance several factors: (1) balancing granularity to capture subtle morphological differences critical for diagnosis, (2) computational efficiency during training/inference, (3) adequate representation across 33 TCGA cancer types, (4) robust statistics within each cluster. Smaller numbers (e.g., 33) lose morphological distinctions; very larger numbers create sparse clusters with insufficient examples.
• Visual Validation: t-SNE visualization reveals clear cluster separation in high-dimensional feature space, while cluster galleries demonstrate both intra-cluster coherence and inter-cluster distinctiveness. Manual pathologist inspection confirmed clusters capture meaningful tissue phenotypes.

This clustering approach has been explained briefly in the main paper (pages 6-7):

• 3.3 Self-Supervised Extension for Unannotated WSIs
• 3.3.1 Tissue Type Discovery via Deep Clustering
• 3.4 Tissue Classifier in Inference Phase

More detailed methodology and analysis have been provided in the Supplementary file (pages 5-9 and 20-29):

• 1.2 Self-Supervised TCGA Clustering Algorithm
• 1.2.1 Clustering Algorithm Details
• 1.2.2 Visualized Cluster Samples
• 1.2.3 Tissue Classifier Training

---

Rare tissue crop selection robustness: Our approach handles rare tissues through several robust mechanisms:

• TCGA Scale Advantage: The massive dataset (634M patches from 11,765 WSIs) ensures comprehensive representation. Rare morphological patterns form distinct clusters in an unsupervised approach rather than being absorbed into dominant categories.
• Adaptive Selection Strategy: Our heterogeneous sampling implements fallback mechanisms: (1) dynamic crop sizing based on tissue complexity, (2) multiple exemplar fusion when single crops are unavailable, (3) quality-aware selection prioritizing high-entropy regions with meaningful tissue diversity.
• Graceful Degradation: Rather than catastrophic failure, the system degrades gradually when optimal crops are unavailable. Semantic maps provide spatial guidance while available crops provide partial morphological information, ensuring consistent generation quality even for challenging scenarios.

More detailed methodology and analysis have been provided in the Supplementary file (pages 2-4, 9, and 10-11):

• 1.1.1 Visual Crop Encoding and Semantic Map Processing
• 1.3 Heterogeneous Patch Sampling Strategy (for TCGA crop and patch sampling)
• 2.1 Visual Crop Size Analysis

Thank you for the positive feedback on our work, particularly noting that the paper is well-written, the comprehensive multi-angle evaluation approach, clear figures, and the impressive 6× FID improvement. We appreciate your constructive comments that help strengthen our contribution.

---

Downstream evaluation data splits: Thank you for this concern about potential data leakage. We implemented rigorous separation protocols to ensure fair evaluation:

• Camelyon16 and PANDA datasets: Diffusion model trained on official training splits. Segmentation model trained on 80% of the official test splits, evaluated on remaining 20% of test splits.
• TCGA: Self-supervised clustering on full dataset, but synthetic generation uses different random crop locations during inference than those seen during training.

This ensures zero overlap between diffusion training data and segmentation evaluation data. The near-parity performance (0.71 vs 0.72 IoU on Camelyon16) demonstrates genuine synthesis quality rather than memorization artifacts. We will clarify this critical methodology prominently in the revision.

---

Mixed real + synthetic evaluation: We evaluated mixed data scenarios and found 2-4% IoU improvements over real-only baselines across both datasets. However, our primary focus on complete data replacement addresses a more fundamental clinical challenge. As stated in our manuscript, "our objective extends beyond data augmentation to complete replacement of real patient data, addressing critical privacy concerns in medical AI development." This capability enables AI development without patient data sharing - particularly crucial for rare cancers and resource-limited institutions where data scarcity is a major barrier.

---

Novelty claims and related work: We appreciate this important clarification. We will:

• Properly cite [1] and acknowledge their application of tissue clustering contribution
• Reframe our novelty claim of the clustering to focus on the self-supervised application and the extension of the clustering approach
• Add comprehensive citations for metadata [2,3] and RNA conditioning [4] approaches in the related work section

Our key technical distinction is using raw pixel crops rather than processed embeddings or metadata. While [1] uses clustering for pseudo-labels (33 clusters matching TCGA subtypes), our approach: (a) scales to 100 morphologically-derived clusters, (b) combines clustering with direct visual crop conditioning, and (c) demonstrates through ablation studies that raw pixel information preserves diagnostic features lost in embedding-based approaches.

---

Visual crop selection during inference: During inference, visual crops are randomly sampled with variable size d×d pixels (d∈[50,200]) from the testing subset that contains segmentation annotations, allowing us to crop correct tissue types (normal/tumor or cluster type) as stated in Algorithm 1 (P.5 L.151). For each tissue class i present in the target semantic map, we: (1) identify spatial regions labeled as class i in the test subset, (2) randomly extract a square crop from within those regions, (3) place this crop at random coordinates within a zero-filled tensor matching full patch dimensions, (4) concatenate semantic maps with these visual crop tensors to create conditioning signal c = concat(M₁,...,Mₖ, C₁,...,Cₖ). This random selection strategy preserves authentic tissue characteristics while providing spatial guidance for synthesis, as described in the main paper (pages 6-7):

• 3.3.2 Adaptive Heterogeneous Region Sampling

More detailed methodology and analysis have been provided in the Supplementary file (pages 2-4, 9, and 10-11):

• 1.1.1 Visual Crop Encoding and Semantic Map Processing
• 1.3 Heterogeneous Patch Sampling Strategy (for TCGA crop and patch sampling)
• 2.1 Visual Crop Size Analysis

---

Minor corrections: We will fix missing citation line 132 and update table nomenclature (H-optimus-0, UNI2-H) as suggested.

We appreciate your thoughtful review and encouraging feedback. It's particularly gratifying that you recognized our comprehensive evaluation approach across multiple datasets and encoders, as well as the pathologist assessment demonstrating superior synthetic data quality. Your acknowledgment of our near-identical segmentation performance and open-source commitment validates our efforts to advance the field.

---

Privacy preservation assessment: Thank you for raising this critical point about our privacy claims. We've implemented a comprehensive quantitative privacy evaluation using the suggested FLD framework and additional analysis to address this critical issue.

• Quantitative Privacy Metrics - FLD Analysis: Following your recommendations, we evaluated our synthetic data (generated by diffusion models trained on Camelyon16 and PANDA) using FLD privacy metrics. FLD scores typically range from 0-100, where lower values indicate better privacy preservation (reduced risk of tracing synthetic samples back to training data). The results demonstrate strong privacy preservation across most foundation model encoders, with our image synthesizing method achieving low FLD values across several encoders.

Table 1: FLD Privacy Analysis Results FLD privacy scores across foundation model encoders for PANDA and Camelyon16 datasets. Lower values indicate stronger privacy preservation, with most encoders showing effective privacy protection.

• Application-Level Privacy Benefits: Beyond synthetic data generation, our framework enables federated deployment where institutions can use our pre-trained TCGA model with their own local visual crops for inference, eliminating the need to share raw patient data.

• Self-Supervised Anonymization: Our TCGA clustering creates morphologically-based tissue phenotypes rather than patient-specific or cancer-type-specific categories. The 100 clusters represent visual similarities discovered through foundation model embeddings, not clinical labels or patient demographics. This inherently anonymizes the training process by focusing on morphological patterns rather than patient-identifiable features.

• Architectural Privacy Protection: Our crop-based conditioning introduces natural privacy protection through spatial fragmentation—no single synthetic image can reconstruct complete patient slides. The random crop placement and size variation (50-200 pixels) further obscure spatial relationships present in original patient data.

---

Results interpretation accuracy: We thank the reviewer for observing this point regarding Table 1 results. We've revised our interpretation to accurately reflect the nuanced performance patterns:

"Prompt conditioning demonstrates substantial improvements on PANDA dataset with consistent 2-6× FD reductions across multiple encoders (GigaPath: 347.3→139.7, RN50-BT: 150.0→22.8). Camelyon16 shows clear improvements with specific encoders (RN50-BT: 430.1→72.0, DINOv2: 122.0→52.7) while TCGA exhibits encoder-dependent performance, with some cases where non-prompt baselines achieve competitive results (H-Optimus-0: 476.0, DINOv2: 117.5, UNI2: 119.6)."

---

Qualitative comparison: Our pathologist evaluation with seven years of clinical experience assessed 120 images across five criteria: image quality, structural detail, nuclear morphology, hallucination presence, and authenticity determination. The expert achieved only 45-52.5% accuracy in distinguishing synthetic from real images (essentially random performance), validating clinical authenticity while confirming privacy preservation through expert-level indistinguishability. A completely dedicated section in the supplementary file (pages 12-15) provides all the details about the synthetic data assessment by an expert pathologist in the section:

• 3. Detailed Expert Evaluation
  • 3.1 Evaluation Protocol and Methodology
  • 3.2 Evaluation Criteria and Clinical Relevance
  • 3.3 Quantitative Results and Statistical Analysis
  • 3.4 Dataset-Specific Observations
  • 3.5 Clinical Implications and Expert Commentary