StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images

My major concern is the significance of the proposed dataset. The main difference between the proposed dataset and previous counterparts is the additional dense open-set spatial semantic maps. The authors claim that "we expect StableSemantics to catalyze advances in visual semantic understanding and provide a foundation for developing more sophisticated and effective visual models". However, the reviewer remains unclear about the usage of semantic maps. How are semantic maps being used in existing model training (e.g., VLMs, MLLMs)? In addition, whether the proposed dataset can actually benefit downstream tasks remains doubtful. More experiments are needed to demonstrate the effectiveness.

My main concerns surround the value add of this dataset in comparison to DiffusionDB and JourneyDB

• the authors claim that StableSemantics differs from datasets like JourneyDB because StableSemantics only collects prompts which are human generated. However, from looking through the JourneyDB collection process it seems that they also collect user prompts from discord servers: is the difference that JourneyDB also contains prompts which are not human generated? If so, this seems like a very small delta.
• the authors show in table 8 that their prompts and captions have a lower perplexity score and thus is more similar to human language than DiffusionDB and JourneyDB, but it is unclear to me why this is desired.
• In figure 7 the authors show that the captions are a subset of JourneyDB's, indicating to me that these captions don't capture a different distribution than existing datasets. This seems like an argument that this dataset provides little value
• there is a lot of mention about the distribution of prompts and captions being specifically objects/styles/scenes that users would find visually appealing. While i appreciate the authors honesty, I do think this reduces the value of this dataset as it only captures a small distribution compared to natural images which are often inputs into captioning and segmentation models.

While an interesting idea, I do not think that this paper has shown that it provides substantial value when compared to existing datasets. That being said, I hope the authors can convince me otherwise by giving some applications of when someone would prefer this dataset over the others. One immediate thing I can think of is to generate the attribution maps for all or a subset of JourneyDB and see if the model lineup changes for the captioning and segementation models. If these differ and the authors show that the evaluation on their dataset provides insights into these models that JourneyDB does not, I would be compelled to improve my score. I am also happy to clarify any of my points in the rebuttal if they were unclear

1. One of the key contributions of this work, relative to prior work, is the inclusion of semantic maps. In L095-L096, the authors write: “Specifically, there are no existing large-scale generated datasets that contribute semantic maps in addition to images and their corresponding captions.” In writing this weakness, I’m interpreting the authors’ claimed contribution to be semantic maps that are similar in usefulness/accuracy to semantic segmentation or instance segmentation annotations. This interpretation comes from two sources: (1) the listed applications in L111-L112 “StableSemantics can be utilized in future research on various vision tasks such as object detection, semantic segmentation, semantically meaningful representation learning, image-inpainting and object removal, object editing, etc.“, (2) the use of the semantic maps to evaluate open-set segmentation models in Figure 14.

   Unfortunately, relative to traditional instance or semantic segmentation annotations, the semantic maps provided are pretty noisy in capturing the spatial extent of objects or "stuff". As such, it is unclear how useful they are. A good example is the “mirror” mask in Figure 5, which does not segment the mirror, and instead encodes the surrounding mirror frame. One consequence of this noise is that it means that downstream evaluations (such as those conducted on open-set segmentation models in Figure 14) don’t seem particularly informative.

   As noted above, I’m listing this point as a weakness on the assumption that the authors intend to frame these semantic maps as being essentially semantic segmentation annotations. It may be that this is a misinterpretation on my part, and they solely wish to claim that they are providing generic semantic attribution maps that don’t aim to capture the spatial extent of objects. This would remove the weakness related to noise, but it brings another problem: it makes the dataset less useful overall.

2. I’m slightly concerned that the generated media is specific to a single model (namely sdxl lightning 4step unet). This may not be an issue - perhaps this model is highly representative of many modern diffusion models. However, it is difficult to ascertain from the paper whether this dataset will immediately go “stale” when new models are released. One possible way to mitigate this would be to generate images from multiple models and demonstrate that similar qualitative insights can be drawn from the data associated with each model.

1- As noted in recent studies [1,2], text encoders in models like CLIP have difficulty preserving detailed relational attributes between objects. Since Stable Diffusion XL also uses a CLIP encoder, there is a high probability that generated images include all requested objects and attributes but fail to capture accurate spatial or functional relationships. For example, when prompted with “a banana with checkered skin,” the model may produce an image with a banana on a checkered background, misrepresenting the intended attribute. This limitation restricts the model’s ability to generate complex scenes that require precise relationships between elements. Given this limitation, a human evaluation step is necessary to filter out images that do not align precisely with their prompts, ensuring better relational consistency in the dataset.

2- The paper relies on CLIP scores and DAAM attribution maps as primary metrics to assess image quality. However, these metrics primarily capture object presence rather than the accuracy of spatial or relational context. To ensure that each generated image reflects the intended relationships and layout specified by the prompt, relation-specific metrics or additional human assessment are recommended. This would enhance the dataset’s value for training models capable of nuanced scene understanding and relational accuracy.

[1] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2022). When and why vision-language models behave like bags-of-words, and what to do about it?. arXiv preprint arXiv:2210.01936.

[2] Hsieh, C. Y., Zhang, J., Ma, Z., Kembhavi, A., & Krishna, R. (2024). Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36.