WinSyn: A High Resolution Testbed for Synthetic Data

Weaknesses

1. Experiments: A main motivation for the dataset is that it can help to address the gap between synthetic and real data, however the paper doesn't really show how the dataset can be used to explain why this gap exists. I would suggest that the authors dive deeper into potential reasons for this gap. Are there features or nuances of real-world images that the synthetic data does not capture (perhaps gradient attribution or some other attribution method can be used to explore this).
2. Originality and contribution: While the paper briefly mentions competing datasets, it doesn't give a clear comparison between these. Understanding how this work differentiates or improves upon existing dataset is crucial for assessing its novelty and significance. There are many synthetic and real dataset on useful real-world domains such as street scenes for autonomous driving (eg. CARLA/SynthIA/GTA5 vs KITTI/cityscapes). A dedicated section comparing the new dataset and procedural model to existing ones in terms of size, quality, versatility, and applications would enhance the paper's depth and relevance.
3. Lack of demonstrating broader applications: The paper loosely mentions applications like superresolution, 3D reconstruction, and generative modeling but does not provide experiments or detailed insights into how the dataset enhances these applications. Detailed experiments or use-cases showing the effectiveness of the dataset in these applications would give readers a tangible understanding of its real-world significance.
4. Addressing the sim-to-real gap: The authors emphasize the need to understand critical differences between real and synthetic data, but no conclusive insights are drawn about this using the dataset. A dedicated analysis or experiment that can help to highlight and understand the differences between the domains would add substantial depth to the paper.

[1] Limited contribution. The proposal of a new dataset including both synthetic and real data is always appreciated; however, the paper does not fully justify its significance over other existing datasets. For instance, with respect to the application of segmentation, there are a wide range of datasets especially on city scenes or driving scenes that have been extensively utilized for the task and even in the context of domain adaptation. In this paper the choice of windows as the domain is properly justified, but it did not answer the question on whether existing datasets can already serve the same purpose (or more). In this case, the significance of the dataset is in question.

[2] Limited evaluation. The paper evaluates the proposed dataset in segmentation tasks on various settings. However a missed opportunity given the availability of both real and simulated splits and the discovery of domain gap in its evaluation, is to test the dataset on domain adaptation methods. The paper could have tested the dataset with existing domain adaptation methods on what the baseline benchmark looks like, potential advantage or challenges of the dataset when compared to other existing datasets in the context of domain adaptation. Otherwise it is difficult to evaluate the impact of the proposed dataset on domain adaptation research, without including actual evaluation results.

[3] Writing. The paper includes lengthy introduction and justification of the choice of windows, in the Introduction section. The text mostly served its purpose. However the writing can be greatly improved. For instance, reorganizing the first paragraph of the Introduction section from 6 points into an organic flow of arguments can be made: the main task the dataset is motivated towards, issues with existing datasets, features of the proposed dataset which significantly differentiates itself from existing ones, and other major features of the proposed dataset. Additionally, the 6 points can overlap: 1st and 4th points both talk about why choosing the domain of windows, and also overlaps with the entirety of the second paragraph. The 6th point is confusing: any dataset comprised of images can be used to extract image features. Finally, additional introduction of Wood et al. before referring to it may benefit the readers in better understanding the last paragraph of the Related Works section and Label Adaptation of Section 6.

Parts of the paper are difficult to follow. The introduction could be streamlined. There is no summary of key contributions and findings/results obtained using the proposed synthetic data.

It is mentioned that in addition to semantic segmentation, dataset may include depth (see Figure 1), but no information about depth or any results using these annotations are provided.

Finally, a summary table comparing properties of the proposed dataset and related datasets is missing.

1. It is difficult to say if window category is representative. Author may need statistic or experimental proof to show why we should choose window to study synthetic to real or generative modeling. It will be good to show that the conclusion we find in this paper can also be applied to more complicated scenes.

2. Some claims in the paper are not verified through experiments. While authors mention generative models and synthetic-to-real in the abstract and introduction, all experiments are on semantic segmentation and we do not have concrete evidence if this dataset can be used for generative modeling or synthetic-to-real domain transfer.

3. The size of the dataset may still be limited. The whole dataset contains less than 100K images of windows, which is still much less than typical dataset used for semantic segmentation (COCO) and generative learning (LION). It is difficult to say if the conclusions from experiments on this dataset can be generalized.