Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning

Thank you for your valuable advice and comments!

Q1: The Improvement

This is because instructions in the general dataset mostly do not contain ambiguity. So, the improvement of Visual-O1 on the general dataset is stable but not as obvious as an ambiguous dataset. However, on the ambiguous dataset, we achieved improvements over the baseline model with LISA: $+344\%$, SoM: $+48\%$, LLaVA: $+149\%$, and GPT-4o: $+57\%$.

Additionally, we find that although traditional CoT can bring some improvements to the ambiguous dataset, it causes a observable performance drop on the general dataset due to the lack of explicit multi-turn visual reasoning and reflection processes. Our Visual-O1 not only brings the largest improvement on the ambiguous dataset but also provides stable improvement on the general dataset, showing the practicality of our method.

Q2: Challenging Case like Cipher

Thanks for your great idea. We provide a challenging case like o1's cipher to demonstrate the effect of Visual-O1:

Inspired by the theory of mind, we design a very challenging case. Refer to the Figure F of the appendix, there is a series of objects on the table: a phone, a water-filled cup, a coffee-filled cup, a pen, and a telephone. The water cup is transparent, clearly showing the transparent liquid inside. All objects have labels with their names, but the label next to the water cup shows "juice". The model will be asked the question: "Lisa poured a glass of water, a porcelain cup of coffee, and prepared a pen, a mobile phone, and an eraser for her friend's visit. Then, Lisa labeled all the items and left an amusing note. But here's something strange for her friend. What does Lisa think it substantially be? A. water B. coffee C. juice D. eraser E. phone".

We compare GPT-4o, GPT-4o + CoT, and our Visual-O1. We find that GPT-4o and GPT-4o + CoT respond to the wrong answer: C. juice, indicating their weak understanding of ambiguous multimodal scenarios.

By reviewing the trajectory of GPT-4o + CoT, we note that GPT-4o + CoT recognized that Lisa incorrectly labeled the water cup but mistakenly believed that Lisa thought the cup contained juice because she placed the label. Given that the cup is transparent in the image, we know this is impossible, meaning that although CoT enhances reasoning abilities, its effect is limited in ambiguous multimodal scenarios.

As shown in the trajectory of Visual-O1, it correctly understood that the question was actually asking what Lisa thought the liquid in the glass cup was. Then, noticing the amusing note left by Lisa on the table, it inferred that Lisa intentionally mislabeled the items. Thus, it realized that Lisa knew the cup contained water and gave the correct answer, A. water, correctly interpreting the ambiguous command in the multimodal context.

Q3: Budget Curve

In the previous version of Table 6 of the paper, we reported an extreme sample on the general dataset to show that even if there are slight fluctuations in the reasoning process, we can still return to an effective reasoning process. Additionally, since the general dataset contains mostly unambiguous instructions, the performance improvement is not as large as that of the ambiguous dataset.

To further demonstrate the stability of our method, we extend the experiment to both ambiguous and general datasets, conducting $10$ experiments with different random seeds. We fixed the random seeds to $40$, $41$, $42$, $43$, $44$, $45$, $46$, $47$, $48$, and $49$. As shown in the tables below, we report the average accuracy of all experiments, as well as their maximum, minimum, and variance.

This demonstrates that our method exhibits good stability on the ambiguous dataset and maintains a positive effect on the general dataset.

For further performance analysis with a budget $N > 3$, please refer to Q6.

(Continued in Authors Response 2/2)

(From the Authors Response 1/2)

Q4: Empirical Solution for General-intelligent Model

The reason we design the empirical experience for the general-intelligent model is that the general-intelligent model has limited text generation and reasoning capabilities, making it difficult to perform instantial experience that requires long text reasoning like a high-intelligent model such as GPT-4o.

However, as shown in Table 3 of the paper, we find that if the general-intelligent model uses GPT-4o to help it perform long text reasoning, it also has the potential to use instantial experience and show promising results. This further proves our Visual-O1's flexibility for models with different levels of intelligence.

Q5: GPU Setting

We use the Nvidia RTX 3090 in all experiments for fair comparisons.

Q6: Performance Saturates of the Budget

The performance decrease observed when the budget $N > 3$ is due to the length of empirical experience prompt becoming longer, making it difficult for the current general-intelligent model to perform image-text understanding effectively. However, this is not an unsolvable problem. By adding an extra step where the model summarizes and simplifies long prompts, the prompt length can be shortened, and we find that the accuracy further improves when the budget $N > 3$:

This means that if the model understands long texts well, our method can achieve increasingly higher results as the budget increases. If the model can only understand shorter texts, a budget of $3$ is sufficient to achieve good results.

Thank you again for your reviewing. Do you have any remaining concerns about our paper or response? We are very looking forward to discussing with you.

Thank you once again for your comments! Are there any remaining concerns about our paper? We would be very happy to address any questions or issues you may have.

(From the Authors Response 1/2)

Q3: Motivation of Instantial Experience

We use instantial experience on the high-intelligent model because it has a faster reasoning speed. It only requires one time of API calling as all reasoning and reflection steps are jointly generated in one response. Plus, it does not need one-time optimization.

Moreover, the high-intelligent model also supports empirical experience. In the Visual-O1$^−$ (SoM) line of Table 3 of the paper, empirical experience is also effective for it. However, due to the lack of specific reasoning for each sample, we observe a slight performance decrease compared to the instantial experience.

Inspired by your comments, we slightly adjust the prompt to allow the reasoning process of instantial experience to generate clear instructions explicitly. We find that implementing this is very simple as it only needs to add a sentence to $p_{\mathrm{syn}}$: Explicitly generate clear instructions within <clear instruction> and </clear instruction> tags before answering. As shown in the table below:

We can see that explicitly generating clear instructions can further enhance the performance. We explain this as explicit clear instruction will help the model to review and better understand the instruction.

Q4: Better Prompt to Keep the Original Purpose

We greatly appreciate your suggestion! We further explore the idea you mentioned. We expand the functionality of response synthesis: instead of directly integrating the clear instruction with the original instruction, we designed a simple prompt to rewrite the clear instruction into a joint instruction guided by the purpose of the original instruction, ensuring keeping the original purpose while integrating as much information from the clear instruction as possible. We are surprised to find that the accuracy further increases:

We explain this as the extra step can keep the original purpose by fixing some misalignment between clear and original instructions. This also aligns with the findings in our paper that the response synthesis is important for Visual-O1.

We appreciate your constructive advice!

Q1: Trend of Performance Improvement

In the previous version of Table 6 of the paper, we reported an extreme sample on the general dataset to show that even if there are slight fluctuations in the reasoning process, we can still return to an effective reasoning process. Additionally, since the general dataset contains mostly unambiguous instructions, the performance improvement is not as large as that of the ambiguous dataset.

To further demonstrate the stability of our method, we extend the experiment to both ambiguous and general datasets, conducting $10$ experiments with different random seeds. We fixed the random seeds to $40$, $41$, $42$, $43$, $44$, $45$, $46$, $47$, $48$, and $49$. As shown in the tables below, we report the average accuracy of all experiments, as well as their maximum, minimum, and variance.

This demonstrates that our method exhibits good stability on the ambiguous dataset and maintains a positive effect on the general dataset.

The performance decrease observed when the budget $N > 3$ is due to the length of empirical experience prompt becoming longer, making it difficult for the current general-intelligent model to perform image-text understanding effectively. However, this is not an unsolvable problem. By adding an extra step where the model summarizes and simplifies long prompts, the prompt length can be shortened, and we find that the accuracy further improves when the budget $N > 3$:

This means that if the model understands long texts well, our method can achieve increasingly higher results as the budget increases. If the model can only understand shorter texts, a budget of $3$ is sufficient to achieve good results.

Q2: Response After Selecting Relevant Instruction

We strongly agree that wrong instruction will affect the answer. Like humans, the model cannot guarantee that clear instructions have the same purpose as the original ambiguous one. Therefore, in our paper, we propose the response synthesis, which combines the original instruction and the clear instruction for the model's reference (see lines 245-250 of the paper), to address this issue.

As shown in Table 5 of the paper, response synthesis can effectively mitigate the occurrence of errors in clear instructions.

Inspired by your advice, we also explored a more proactive way to retain the purpose of the original instruction. Please refer to Q4.

(Continued in Authors Response 2/2)

Thanks authors for the response! The significance test results, the summarization method under large budgets, and the Instantial + Explicit Clear Instruction method have significantly improved the paper, so I'm increasing the rating.

Thanks for your invaluable advice!

Q1: Analysis of Failure Cases

We have now added an error analysis. We find that these errors can generally be divided into two categories:

• Errors caused by non-reasoning factors such as calculation, color recognition, and orientation identification errors.
• Errors caused by excessive reasoning.

The first type of error is shown in the left part of Figure E of the appendix, we find that although Visual-O1 helps the model perform reasoning analysis well and generates clear instructions, the model is still limited by its segmentation capabilities, resulting in incorrect outcomes. However, we believe that as the model's understanding ability improves, such errors will gradually decrease.

For the second type of error, we find that sometimes Visual-O1 leads to over-reasoning. As shown in the right part of Figure E of the appendix, we find that Visual-O1 over-interprets the meaning of "reflecting light one," thinking it refers to a small reflective area on the car rather than the whole car. However, the instruction refers to the entire car. Despite the presence of excessive reasoning in some samples, the overall better performance on the general dataset means that, in most cases, Visual-O1 can effectively reason out the correct meaning of the instructions.

Q2: The Improvement

We conduct experiments on both ambiguous and general datasets. The improvement of Visual-O1 on the general dataset is stable but not as obvious as an ambiguous dataset. However, on the ambiguous dataset, we achieved improvements over the baseline model with LISA: $+344\%$, SoM: $+48\%$, LLaVA: $+149\%$, and GPT-4o: $+57\%$.

Q3: Terms of Writing

Thank you for your suggestions. We have revised our statements and will be more cautious about this in the future.

Q4: Average Results over Different Random Seeds

Thanks for your suggestions. We now report the average results of experiments on $10$ random seeds ($40$, $41$, $42$, $43$, $44$, $45$, $46$, $47$, $48$, $49$) and conducted statistical tests.

We find that the improvement is statistically significant with $p < 0.01$ under $t$-test, indicating that Visual-O1 has good stability and generalizability.

Q5: Typos

Thank you for your reminder. We have revised Eq. (6). In Figure 3 of the paper, we show the original instruction, which may contain some typos from the dataset, where "bottle" is misspelled as "botlle." To make this clearer, we have noted this in the caption.

(Continued in Authors Response 2/2)

(From the Authors Response 1/2)

Q6: Difficulty of the Ambiguous Dataset

We further demonstrate the difficulty of our dataset from two perspectives:

(1) Our dataset is relatively easy for humans but difficult for AI models.

We conduct a human experiment by inviting $10$ volunteers to answer questions of our ambiguous dataset. The results are shown in the table below:

We find that humans can correctly answer more than $80\%$ of the questions, indicating that the ambiguous dataset does not pose major challenges for humans. This is because humans have strong disambiguation abilities, enabling them to accurately understand the meaning of ambiguous instructions by combining them with images, a capability that AI models lack.

(2) The difficulty of our dataset lies in the image-text ambiguity.

We design another human experiment where volunteers can only read original instructions but are not allowed to see images. In each turn, volunteers are allowed to ask one question and receive one answer from either (a) GPT-4o, which can see the images but not the instructions, or (b) another volunteer who can also see the images but not the instructions. Each sample will go through up to $10$ turns of Q&A or until the correct answer is obtained. Volunteers cannot ask questions that have the same meaning as the original instructions. We conduct the experiment on both the general dataset and the ambiguous dataset and calculate the average number of Q&A turns for samples that successfully obtained the correct answer:

We find that on our ambiguous dataset, the turns of Q&A greatly improved. This is because, for samples with ambiguity, volunteers have difficulty determining what exactly to ask, leading to numerous tentative questions before finally pinpointing the intended question and obtaining the correct answer. In contrast, on the general dataset, volunteers can often quickly identify what needs to be asked, resulting in much lower turns of Q&A. This also demonstrates that text-image ambiguity in our dataset increases the difficulty of reasoning.

Q7: Implementation of Baselines

For FuDD, the model will find the five relevant objects of ambiguous instructions in the image. Then, the model will generate the key distinguishing information for each pair of objects. The model will use this disambiguating information appending to the question to obtain a better answer. We can find that FuDD focuses more on the ambiguity of similar language and visual content than explicitly considering the ambiguity that requires reasoning with visual context.

For CoT, we use the classic prompt: "Think it step by step." and our task-modified prompt: "Analyze the question based on the image." There are slight differences in performance of two prompts and we report the higher scores in our experiments. We can find that CoT focuses more on reasoning at the language level but lacks the explicit process of reflection on multi-turn and multi-modal reasoning.

Q8: Gray Sentence and Trajectory

Yes, the sentences in gray are given by reflection. In practice, models sometimes do not engage in reflection at every step. We have observed that in some highly certain steps, the model tends to skip reflection, which also helps to speed up the reasoning process.

We also provide two additional groups of raw trajectory: one for a normal case and one for a challenging case, compared with CoT without any highlighting, as shown in Figures C and F of the appendix.

Thanks again for your comments! Is there any remaining concerns about our paper? We are more than delighted to address any concerns/questions you may have.