Benchmarking and Analyzing Monocular Geometry Estimation Models

1. The paper lacks clarity and is hard to read. I was wondering actively why after each sentence I read I had problems remembering what I just read. The result of my analysis: the sentences use strange or vague terminology with insufficient context and were not to the point. The same is true for whole paragraphs. Information is spread all over the place without any help given to the reader to make sense of it. This makes it very tedious to read this paper. It was no fun reading it. I did it because I had to.

2. There are only vague formulations when it comes to the contributions. For example, it is claimed that "we build fair and strong baselines", but when reading on, the baselines are simply existing models that have been finetuned on a different data subset. This indeed allows a more direct comparison, but nothing new is built here. Another example on benchmarks: "we build more diverse scenes with high-quality labels for geometry evaluation. For depth estimation, we introduce three extra benchmark datasets", but in fact these datasets already exist and were simply used for the experiments. Again, nothing new is built here.

3. It would be okay with nothing new being built, if this paper had a strong scientific side. There was some hope from the abstract, which talks about many insights. However, the insights are anecdotal and only mildly conclusive. Except one attribution to decoder supervision, there is no analysis of the causes of a certain behavior. Therefore, the findings can be due to multiple reasons not controlled in the experiments. As a result, there is nothing conclusive I can take home. The conclusion by the authors "We identify that a strong pre-train model, either Stable Diffusion or DINOv2, combined with high-quality fine-tuning data, is the key to achieving generalizable monocular geometry estimation." doesn't really tell anything. Only Stable Diffusion and DINOv2 were tested, so it is impossible to identify that they are really important, and that high-quality fine-tuning data (what does high-quality mean exactly?) is important for good results is not a new finding.

1. The article resembles an experimental report more than a formal academic paper. While it thoroughly documents the comparisons, it lacks the depth and structure typically expected in scholarly research.

2. Lack of novelty. Although the paper offers valuable recommendations, it falls short of proposing specific solutions or implementing them, which limits its contribution to advancing methodology.

3. While the paper appears to aim at establishing a new benchmark, it does not introduce a new dataset but rather relies on existing datasets. This reliance on pre-existing data weakens the contribution toward setting a truly novel benchmark, as it doesn’t address potential limitations or gaps within current datasets that new benchmarks typically seek to fill.

• Lack of Depth in Analysis: The paper often bases its conclusions on somewhat limited experimental results. For example, it concludes that complex fine-tuning protocols are not necessary based solely on the strong performance of GenPercept's one-step fine-tuning. However, this could be a risky generalization. This conclusion might easily be challenged if a more effective, complex fine-tuning method is proposed in the future. Rather than relying solely on performance metrics, a more in-depth analysis of GenPercept’s network architecture and training dynamics is needed, along with validation under diverse experimental conditions. For instance, evaluating the robustness of different types of datasets or under significant viewpoint changes could enhance the reliability of this conclusion.

• Novelty: Although the paper provides important insights through extensive experimentation, these insights do not introduce entirely new directions. The study effectively aggregates and compares recent research to showcase the current leading approaches. However, this approach leans more towards a re-evaluation of existing work rather than proposing novel methodologies or technological breakthroughs. Therefore, it is questionable whether such comparisons alone are sufficient as substantial contributions.

W1: L419 'However, the performance of small-scale dataset finetuning is largely behind the official model trained on 16M data samples. Hence, large-scale datasets with diverse scales and cameras is still indispensable for metric depth estimation.' This conclusion is not convincing enough, since the original Metric3D V2 has been trained for 800k iterations with a batch of 192, while for the retrained model, the training iteration is 30k with unknown batch number (maybe 96 mentioned in L397?).

W2: About the evaluation metrics. Since the authors pay attention to the current zero-shot depth estimation and normal estimation methods, which pay much attention to 'robustness', maybe a metric about the error variance is needed to represent the robustness of each model on the benchmark. For example, now the AbsRel and delta_1 are all the mean values, maybe you can present the variance of AbsRel and delta_1 to present the model robustness.

W3: the overall performance is needed. Since most of the Tables are evaluated through several datasets/benchmarks, a overall result in each table is needed to present the model performance.

W4: most of the experiments are conducted under insufficient training (20k iterations), which may affect the validity of the experimental conclusions.