PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration

Some SOTA pathology foundation models are missing, especially in experimental comparison, such as UNI [1], CONCH[2], mSTAR[3] (which is also a CLIP-style model trained on TCGA data as well), etc.

Thank you for your suggestion. As UNI is a vision-only model and mSTAR has not yet been open-sourced, we focused on comparing our method with more similar model, CONCH. This comparison includes zero-shot classification, few-shot classification, and WSI classification tasks. PathGen-CLIP-L consistently demonstrates better performance than CONCH across these tasks. For detailed results, please kindly refer to the General Response.

Few-shot’s capability significantly relies on how well-aligned the vision and text spaces are. The proposed method is supposed to compare with VLM like CONCH, instead of vision-only GigaPath.

Thank you for your suggestion. We provide a detailed comparison of the few-shot capabilities between our proposed method and CONCH in the General Response. Please kindly refer to the General Response for further details.

Details in EXPERT EVALUATION are missing. For example, how do authors define what correct or incorrect findings are?

Thank you for your comment. In our expert evaluation, findings were assessed in a straightforward manner based on two criteria: (1) whether the description aligns with observable image features (e.g., stating there is necrosis when only scattered mucin is visible would be incorrect); and (2) whether the description uses standard professional pathological terminology (rare cases where pathologists could not understand the findings were deemed incorrect).

In Step 5 of data construction, how do authors ensure that no essential details are lost?

In step 5, the description agent's outputs tend to be verbose, and the summary agent focuses on condensing these overly detailed sentences into concise summaries without losing essential information. Additionally, during the prompt engineering for GPT-4, we actively incorporated feedback from pathologists, iteratively refining the prompts until they produced results that met their satisfaction. Occasionally, certain details, such as color or HE staining information, may be omitted during this process. However, these details are not critical for accurate diagnosis and do not impact the overall effectiveness of the findings.

How can be validated the effectiveness of each step in data curation? This should be discussed in the ablation study.

Thank you for your insightful feedback. Following your suggestion, we conducted an ablation study to validate the effectiveness of each step in our data curation process. As illustrated in following table, removing either the clustering-based retrieval method or the prompt-based retrieval both led to a decline in performance.

• Notably, the removal of the clustering-based retrieval resulted in a more substantial performance decrease of 4.3%. We hypothesize that this is because clustering-based retrieval is capable of sampling patches with more diverse features, thereby enhancing the overall dataset quality.
• Additionally, eliminating the revise agent caused an average performance drop of 2.3%, indicating its effectiveness in correcting errors or hallucinations generated by LMMs.
• These findings confirm that each component of our data curation pipeline is crucial.

We sincerely thank you for your valuable suggestions and recognition of our work. Please refer to our responses to your suggestions below.

The performance of directly utilizing original report data along with patch images to pretrain a pathology foundation model should be presented.

Thanks for the suggestion. On one hand, WSI reports provide a global summary and cannot accurately represent the features of each local patch. This mismatch between patch-level features and shared captions introduces significant challenges for CLIP's contrastive learning, as the model struggles to establish meaningful associations between image and text embeddings.

Moreover, the dataset contains only about 7,500 WSI reports, which further limits the model's ability to learn more diverse features. To illustrate this point, we trained PathGen-CLIP-L by using the WSI report as the caption for every patch in the WSI. As shown in the following table, the model trained in this manner performs even worse than PathGen-CLIP-L_init.

To validate the extra contribution of generated patch descriptions, the proposed model should be compared with the one trained on these selected representative patches as well as its corresponding report data.

We sincerely appreciate your suggestion, and in response, we conducted additional DINO-v2 pretraining on the images from PathGen-1.6M. As shown in the following tables, DINO-v2 pretrained on PathGen-1.6M images significantly underperforms PathGen-CLIP-L in few-shot experiments. This is likely because DINO-v2 requires substantially larger samples—on the scale of tens or even hundreds of millions of images—to achieve optimal performance. In contrast, this highlights that the high-quality image-text pairs in PathGen-1.6M are key to enhancing the model's capabilities, serving as a more effective approach than vision-only self-supervised learning approach like DINO.

Dear Reviewer 33XD,

Thank you for taking the time to read our response. We are glad to hear that the majority of your concerns have been addressed.

Regarding your concern about the validation of the extra contribution of generated patch descriptions: In our first response, we conducted experiments to demonstrate that PathGen-CLIP-L trained on the selected representative patches and their ''paired reports'' leads to significant performance degradation (-10.6%) compared to PathGen-CLIP-L trained on the generated captions (we attach the table below for reference).

This performance degradation occurs because the patches themselves do not have corresponding reports. Instead, the WSI, which contains thousands of patches, is paired with a single report. By attempting to pair individual patches with the WSI report, it will introduce significant misalignment and a many-to-one relationship, which adversely affects the contrastive learning in CLIP.

For the subsequent DINO experiments, we conducted additional experiments to validate the effectiveness of PathGen-1.6M from a different perspective (image-only) to further demonstrate the improvement is primary from PathGen-1.6M's image-caption pairs.

We hope this explanation addresses your concern, and we look forward to your reply. If we have misunderstood your question in any way, we would be more than happy to conduct further experiments to validate your concern.

Thank you once again!

Don't compare against better publicly available pathology vision encoders like H-Optimus.

Thank you for the suggestion. Our main contribution is the PathGen-1.6M dataset per se, which is why we chose datasets and benchmarks as our Primary Area. The PathGen-1.6M dataset can be effectively applied to more advanced models (such as GigaPath and H-Optimus) to further enhance performance. As highlighted in Section 4.6, we demonstrated how PathGen-1.6M could fine-tune vision-only models like GigaPath, achieving the following capabilities:

1. The transformation of vision-only models into vision-language models, enabling multimodal capabilities.
2. Further improving their vision performance.

Additionally, our model was trained solely on publicly available datasets, specifically 7,500 WSIs from TCGA, using significantly fewer patches and smaller hyperparameters (OpenAI CLIP-L). In contrast, models like H-Optimus were trained on over 500,000 privately collected WSIs, utilizing 4 times larger model VIG-G. Such data scale and resources are inaccessible to researchers relying solely on public datasets.

There isnt a comparison of the performance of the original PathGen-CLIP-L_init & Comparison which would be useful is understanding with/out data generated using the revision agent.

Thank you for the suggestion. We have included the results for PathGen-CLIP-L_init, and PathGen-CLIP-L significantly outperforms it, with an average improvement of 6.8%. Notably, performance improved by 14.9% on SkinCancer, 17.7% on CRC, and 15.2% on SICAPv2. This highlights that PathGen-1.6M can significantly enhance the image-text alignment capabilities of CLIP.

Additionally, without the Revise Agent, the performance of PathGen-CLIP-L suffers a 2.3% degradation. This demonstrates that the Revise Agent effectively corrects potential errors or hallucinations from the description agent.

Its also unclear how important the scale of the detailed caption dataset 30K used for training PathGenLlava and the quality of it. Does scaling the dataset size and having some refinement here help?

Your suggestion is absolutely correct. We could investigate scaling up the detailed caption dataset to explore whether training a better description LMM agent could further improve model performance. However, the challenge lies in the fact that each time we scale up the detailed caption dataset, we need to retrain and regenerate almost all the models and data, which results in substantial time and cost overhead. We apologize that the rebuttal period was too short for us to conduct this experiment and we plan to address this in future work.

However, to demonstrate that improving the quality of the dataset is effective from another perspective, we used our trained PathGen-LLaVA model to refine the captions of the 700K PathGen_init dataset. We observed that this refinement can lead to a 1.4% improvement in model performance. This indicates that a higher-quality image-caption dataset can enhance overall model performance.

Thank you for your valuable and insightful feedback, which has greatly helped us refine and improve our work. We have carefully addressed each of your suggestions one by one in our response.

Its unclear why and how the subset of 700K samples was choosen from existing datasets to create PathGen_init.

This is an excellent question. The performance of prompt-based retrieval and integration with LLMs heavily depends on having a well-performing CLIP model. However, existing models are constrained by the quality and quantity of their training data, resulting in suboptimal multimodal capabilities. To overcome this limitation, we carefully curated and collected a larger vision-language dataset of 700K samples to train a more robust CLIP initial model. As shown in the table below, the PathGen-CLIP-L_init model, trained on this dataset, already surpasses the previous SOTA models.

In constructing PathGen_init, we retained almost all of the data from PathCap, particularly as the authors had already performed some level of data cleaning. For the Quilt-1M and OpenPath datasets, we conducted an additional cleaning process to address instances where captions did not align well with the images. For example, captions in social media posts often emphasized the aesthetic qualities of the images or solicited diagnostic opinions from the public. Furthermore, there was some redundancy among the images. After this cleaning process, we preserved a total of 700K high-quality data points.

For training the description agent, the authors mention using 10K initial image-caption pairs to generate 30K dataset, so are 3 captions generated per image?

We apologize if our wording caused any misunderstanding. As we mentioned, “we sample 10,000 image-caption pairs from PathCap, OpenPath, and Quilt-1M, respectively,” this means that 10K samples were taken from each of PathCap, OpenPath, and Quilt-1M, adding up to a total of 30K. We will revise our wording to make this clearer.

Revision Agent: What model is used for training the revision agent? Its also unclear what the inputs for revision agent are at inference? Does it take the generated caption and produce possible edits?

We apologize for not expressing this clearly. Revise agent is trained using the LLaVA v1.5 framework. During the inference phase, the input consists of the description generated by a previous LMM description agent, paired with the corresponding image, forming a multimodal input. Revise agent processes this input to produce a series of edits.

In figure 10, its unclear the w/ PathGen_init two stage performance doesnt vary when scale of PathGen data is varied? When scale is 0 it means all data is PathGen_init is that correct?

We apologize for not explaining this more clearly. The two-stage approach with PathGen_init serves as a reference point, representing the final result of our current method, which is why its performance does not vary. For the orange line (with PathGen_init one-stage), a scale of 0 indeed means that all the data is from PathGen_init.

Thank you for pointing this out, and we will include a more detailed explanation in the appendix.

Its unclear why they couldn't get access to the more recent CONCH which is publicly available on HuggingFace.

Access to CONCH requires approval, and since we obtained access relatively late, we were unable to include experiments involving CONCH. However, we have now included experiments to benchmark CONCH in General Response of the rebuttal. PathGen-CLIP-L demonstrates significantly better performance across various downstream tasks compared to the CONCH. Please refer to the general response for more details.

Access to CONCH requires approval, and since we obtained access relatively late, we were unable to include experiments involving CONCH.

Dear Reviewer TdEJ,

Thank you once again for your time and thoughtful consideration. We hope that we have addressed your concerns, and would be grateful if you could share your feedback. We are more than happy to provide any further clarification or engage in additional discussion if needed.

Dear Reviewer TdEJ,

Thank you very much for taking the time to read our response, and we also appreciate your further suggestions.

Regarding the mention of data cleaning, we have made updates in lines 149-152, which are highlighted in red in the manuscript. These cleaned datasets will be made fully public. However, since we do not directly own the rights to distribute these datasets, we will provide the image IDs, and researchers can extract the images from the original datasets.

As for the additional experiments, such as the comparison to PathGen-CLIP-L$_{init}$, we have included these results in Appendix section B5.1, Table 9, along with some other ablations.

If you feel that our responses resolve the concerns raised, we would be grateful if you could kindly consider this in your evaluation. Please feel free to let us know if you have any further questions or would like additional clarifications.

Thank you once again for your time and reply!

We deeply appreciate your thoughtful and constructive feedback. We have carefully addressed each of your concerns in detail and incorporated your suggestions to strengthen our manuscript.

It seems ineffective in addressing common pathological inaccuracies in the generated descriptions.

Thank you for your thoughtful question. The Revise Agent is specifically designed to address certain inaccuracies or potential hallucinations in the generated descriptions. For instance, it can help correct errors such as mistaking scattered mucus for necrosis or overdiagnosing inflammatory regions as cancer. While it may not completely eliminate all inaccuracies, it does provide a significant improvement compared to making no corrections at all.

To evaluate the effectiveness of the Revise Agent, we conducted a benchmark comparing the zero-shot performance of CLIP with and without the Revise Agent. The results indicate that the Revise Agent improves overall performance by 2.3%. This demonstrates its ability to enhance data quality by generating more accurate captions, which in turn strengthens the model's capabilities.

The writing of the paper requires improvement. For example, there is a typo in “PathGen-LLaVAdesp” in Section 4.1, and the conclusion summarizes, “we train two advanced models: PathGen-CLIP and PathGen-CLIP-L.” It appears there is an additional model, “PathGen-LLaVA,” which needs clarification.

We apologize for any potential confusion caused, as there are many intermediate agents and the final trained model, which might have led to some misunderstanding. Specifically, PathGen-LLaVAdesp is not our final model but rather an intermediate model used exclusively for generating image descriptions.

Overall, our final output models are of two types: a CLIP series (PathGen-CLIP and PathGen-CLIP-L, trained using the PathGen-1.6M image-caption dataset) and a large multimodal model (PathGen-LLaVA, trained using PathGen-instruct 200k, capable of handling instruction-following tasks such as visual question answering, classification, and captioning).

The sentence we mentioned, "we train two advanced models: PathGen-CLIP and PathGen-CLIP-L," it specifically referred to the two different sizes of the CLIP versions.

A significant concern arises regarding potential data leakage in the downstream tasks and benchmarks evaluated. To my knowledge, many tasks and benchmarks are derived from TCIA pathology data, which raises suspicions about the homogeneity of the constructed dataset.

Thank you for your valuable feedback. To address concerns about data leakage and ensure dataset diversity, we have taken deliberate measures to avoid overlap with TCGA/TCIA subtyping datasets, such as TCGA-BRCA and TCGA-RCC, in our downstream tasks. Instead, we selected datasets like BRACS (a different dataset from TCGA-BRCA), Camelyon17, Camelyon16, and PathMMU, none of which are derived from TCGA/TCIA.

This careful selection minimizes homogeneity and ensures the robustness and generalizability of our results. Below, we provide the source links for each dataset for your reference:

• PatchCamelyon17: https://patchcamelyon.grand-challenge.org/Download/
• CRC-100K: https://zenodo.org/records/1214456
• SICAPv2: https://data.mendeley.com/datasets/9xxm58dvs3/1
• BACH: https://iciar2018-challenge.grand-challenge.org/Dataset/
• Osteo: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0210706
• SkinCancer: https://heidata.uni-heidelberg.de/dataset.xhtml?persistentId=doi:10.11588/data/7QCR8S
• MHIST: https://bmirds.github.io/MHIST
• WSSS4LUAD: https://wsss4luad.grand-challenge.org/
• LC25000 (LC-Lung and LC-Colon): https://github.com/tampapath/lung_colon_image_set?tab=readme-ov-file
• BRCAS: https://www.bracs.icar.cnr.it/
• Camelyon17: https://camelyon17.grand-challenge.org/Data/
• Camelyon16: https://camelyon16.grand-challenge.org/Data/
• PathMMU: https://pathmmu-benchmark.github.io/#/

Dear Reviewer Y3n6,

Thank you for your time and thoughtful consideration. We sincerely hope that we have addressed your concerns adequately. If there are any remaining issues or if you require further clarification, please do not hesitate to let us know.

Once again, we truly appreciate your valuable feedback and support.

Details on how the instruction tuning data was curated are not provided

Thanks for pointing is out. We apologize for not clearly explaining the process of generating the instruction-tuning data. Specifically, we randomly sampled 95K and 105K non-overlapping examples and used the prompts (shown in Figure 15 and Figure 16 in the appendix) to prompt GPT-4 to generate instruction-tuning data based on the sampled image-caption pairs. We will clarify this in the main paper in the future revision.

While not the goal of the paper, it will help to include fully supervised baselines as well to educate the readers of the gap with CLIP like models

Thank you for the suggestion. We provided fully supervised baselines as follows.

• As shown, there remains a significant gap between zero-shot performance and full fine-tuning. However, the advancements of PathGen-CLIP-L bring the performance of pathology-specific CLIP closer to that of full fine-tuning.
• Moreover, the full fine-tuning performance of PathGen-CLIP-L shows a substantial improvement over OpenAI-CLIP-L and is comparable to GigaPath, despite the latter benefiting from significantly larger training datasets and models.

Referring to this caption generation system as agent-based might be somewhat exaggerated.

Thank you for your valuable feedback. While we acknowledge that our system currently functions more like a sequential pipeline, it can still be conceptualized as a simple agent-based framework, where each module operates autonomously within its defined scope (e.g., the Revise Agent can independently decide whether to modify captions based on the received image-text input). Similar modular designs have been employed in works like MORA [1]. We believe that expanding components, such as the Revise Agent or LMM Agent, into more interactive roles—such as "round-table discussions"—would further enhance the system’s agent-based nature, and we leave this for future work.

[1] Yuan Z, Liu Y, Cao Y, et al. Mora: Enabling generalist video generation via a multi-agent framework[J]. arXiv preprint arXiv:2403.13248, 2024.

Thank you for your thorough review and detailed suggestions. We appreciate your recognition of the strengths of our approach. Regarding your questions, we address them one by one below:

How many real pathology report findings did you'll extract in section 3 and how did you verify the quality of it

Thank you for your feedback. To clarify, the pathology report findings for WSI were extracted as 1-3 blocks of paragraphs containing the findings, rather than segmented into independent entries. This is because the entire report often exceeds CLIP's 77-token limit, so the portions of the report that exceeded the length were divided into two to three sub-paragraphs to ensure all relevant information was retained without truncation.

To ensure quality, we conducted multiple rounds of prompt engineering. In each iteration, a human pathologist reviewed at least ten reports cleaned by GPT-4o. This iterative process involved 3-4 rounds of refinement and expert feedback, culminating in a robust prompt that enabled GPT-4o to generate cleaned reports closely aligning with pathologists' expectations.

How do you evaluate the performance of the images retrieved for the prompts in section 3.1?

We involved pathologists to manually evaluate the retrieved image patches to ensure they met expectations. Specifically,

• Pathologists randomly reviewed 3 WSIs (as reviewing a large number of patches is highly time-consuming, it was not feasible to evaluate all WSIs).
• They first identified observable findings in the predefined prompts and evaluated whether the retrieved patches captured these findings.
• The evaluation revealed that 71.1% of the patches matched the findings, when representative patches from clustering-based retrieval were included, this percentage increased to 80.3%.

How many prompts do you use for each WSI?

For each WSI, we generate 1-3 findings-based prompts and 20 attribute-based prompts, resulting in a total of 21-23 prompts per WSI.

How did you arrive at the design choice of using both prompt-based and clustering-based retrieval?

This is a very good question. We combined prompt-based and clustering-based retrieval to balance diversity and representativeness in the retrieved patches. Prompt-based retrieval utilizes high-quality WSI reports to focus on diagnostically critical patches, while clustering-based retrieval ensures semantic diversity by capturing a variety of features across the WSI. This combination also addresses individual limitations, such as the prompt-based method's occasional tendency to focus entirely on irrelevant artifacts like doctor-annotated black markings on WSIs. Clustering-based retrieval can avoid this by retrieving other relevant patches.

Why chose a threshold of 0.88 for similarity?

This is an empirical result achieved by inviting pathologists to evaluate different thresholds. We adjusted the threshold to ensure the filtered images maintained appropriate distinctiveness.

Is similarity computed globally in all the extracted patches across slides?

Similarity is calculated within a single WSI. This approach is chosen because patches within the same WSI are more similar compared to patches across different WSIs. Computing similarity globally across slides could risk over-deleting certain samples.

Does the revision agent have a no-operation capability as well? What happens when a correct description is passed to the revision agent?

Yes, absolutely. During training, we retained approximately one-third of the data without any modifications. The revision agent outputs a list of operations, and when a correct description is passed to the revision agent, its output is simply an empty list.

What does first and second stage training in PathGen-CLIP mean?

The first stage involves pretraining PathGen-CLIP using only PathGen-1.6M, followed by fine-tuning in the second stage with PathGen_init, which yields the best performance. For a detailed explanation of this approach, please kindly refer to Appendix Section B.3.

Why did you use different datasets for zero and few shot experiments? As an example, Camelyon is not included in zero-shot examples

Thank you for pointing this out. Since datasets like BACH and SkinCancer cannot provide the full 256 shots per class due to their size, and given space constraints in the article, we only included results for four datasets.

Here, we present the complete few-shot results for the remaining five datasets, using the largest feasible number of shots (2, 4, 8, 16, 32, 64, 128, and 256) based on the dataset size. As shown in the following tables, the PathGen-CLIP series demonstrates significantly better performance compared to previous CLIP-based pathology models.

Dear Reviewer wDqp,

We hope that we have addressed your concerns, and would be grateful if you could share your feedback. We are more than happy to provide any further clarification or engage in additional discussion if needed.

Many thanks for your time and effort!