Re-Aligning Language to Visual Objects with an Agentic Workflow

Dear Reviewer M1hG,

We would like to thank you for providing valuable comments, and we answer the concerns raised below. We have included all discussion results in our revised manuscript (content in orange color).

Choice of VLM: The authors use the commonly utilized large-parameter version of VLM (LLaVAv1.6-34B). However, several newer Controllable VLMs / MLLM / Captioner allow interaction through a variety of visual prompts (such as Points, Boxes, and Masks) to obtain more focused descriptions for target regions. Directly using these controllable VLMs may have fewer hallucinations on region caption task and may be better than LLaVAv1.6-34B with region cropping. Examples Region Controllable VLMs include: AlphaCLIP (CVPR 2024) [1], Osprey (CVPR 2024) [2], Ferret (ICLR 2024) [3], OMG-LLaVA (NeurIPS 2024) [4] and so on.

We appreciate the valuable advice. We observe that controllable VLMs may have hallucinations on region caption tasks like LLaVAv1.6-34B with region cropping. Our proposed method is a generalized workflow schema compatible with these different VLM tool models and various visual prompts (such as Points, Boxes, and Masks). The goal of these visual prompts is the same as our visual prompts updated by image editing operations (especially the object highlight operation) to help VLM focus on different ranges of target regions. In addition, the adaptive customization of text and visual prompts generated by the agent is one of the core contributions of our workflow that can also help different VLMs perform better. We chose the LLaVAv1.6-34B and those simple image editing operations because they are widely used and can better demonstrate the generality of our method. These strong VLMs with better detail reasoning capabilities encourage us to explore their ability further to help our workflow deal with more complex cases in the future. We have cited them in Ln 45-47 to show our respect.

Lacking ablation experiments on the influence of the fine-tuning of ChatGLM. What is the role of this step? Can fine-tuning enhance the effect of all three stages of the Agent.

The role of fine-tuning is to adapt ChatGLM into a specific agent that can plan states/actions and output results in a desired format in our workflow for VL (vision-language) data alignment while preserving the reasoning ability. The fine-tuning strongly enhances the effect of all stages of the agent. When directly using the pure ChatGLM, the planning process is uncontrollable and can not reason for the desired output in a specific format, which our workflow can not parse. This drops the success rate to a low degree, nearly the success rate of random selection schema, as shown in Sec. 3.3.

Error analysis about the workflow of agent. Lack of analysis of Agent workflow failure cases. To which part of the Agent should the cause of the data refinement error be attributed: inference? Tool call? Error perception of the tool itself? Error of reflection?

Thanks for pointing out the lack of analysis of failure cases. It has a high guiding significance for further optimization of Real-LOD. We look into the failure cases and find out there are two main kinds of data refinement error caused by the error perception of the tool model (VLM):

1. Typical visual hard cases. As illustrated in Sec I.2 of the appendix, object-detection datasets include low-light or low-quality scenes and extremely small or difficult-to-recognize objects. It could be difficult for VLM to generate appropriate expressions for these targets.
2. Expression describes a foreground object instead of the target background object. VLM may ignore the target object in the background when an occlusion exists. In order to ensure high quality at the bbox level, the reflection module regards these expressions as wrong.

When conducting large-scale data refinement, we set the maximum iteration to 4. With several extra iterations, some failure cases are likely to be solved. Since we already have a large amount of high-quality data for downstream training with a task-solved rate of 75%, we choose not to increase this parameter for the sake of efficiency.

Revised contents:

1. We have included visualizations and analysis of failure cases in Sec. I.2 of the appendix.

Dear Reviewer e1U7,

We would like to thank you for providing valuable comments, and we answer the concerns raised below. We have included all discussion results in our revised manuscript (content in orange color).

• Authors use an agentic workflow controlled by an LLM to reduce VLM hallucinations. However, there is still hallucinations in the LLM. In fact, hallucinations are not effectively reduced.
• In Section 3.2, there are "Right" state and "Wrong". How to judge whether he is truly right or wrong, rather than the hallucinations from the LLM?

We agree that there are still hallucinations in the LLM. However, we have conducted the following schemes to help the agent judge whether the expression is truly right or wrong (i.e, VLM hallucination), rather than the hallucinations from the LLM:

1. We introduce the ground truth category label from the detection dataset. This label serves as a strong reference during reflection. Our Real-LOD can identify the expression that contradicts the label as a VLM hallucination (i.e., truly wrong).
2. We observe that different foundation models (i.e., Vicuna, Llava) in our workflow normally do not encounter the same hallucinations. Therefore, we use them to interact, examine, and improve expressions iteratively within the workflow. This enables the Real-LOD to judge whether the current expression is truly wrong from diverse perspectives, advancing its robustness and reducing hallucinations from a single model. Meanwhile, when model discrepancy still exists once reaching the max iteration number, we regard this as a hallucination and can directly discard this expression to preserve training data quality. Besides, our specifically designed prompts for these models also further reduce their own potential hallucinations.
3. We introduce an LLM-based reflector in each iteration to analyze the correctness of the current expression based on the given reference information. We observed that the reflector never introduces new errors caused by LLM hallucinations and has a high recall for expressions that may have a VLM hallucination (i.e., Uncertain/Wrong states). This is because text analysis is the primary strength of LLMs, and the reflection task is straightforward. Therefore, its feedback to the agent can help Real-LOD further reduce the VLM hallucinations in a given expression.

With the above schemes, the hallucinations are effectively reduced, demonstrated by the improvement of the evaluation results in Sec. 4 and examples in Fig. 5 and Sec. D.

The general LOD framework in Figure 3 is very common. Thus, I have concern about the technological novelty of the presented method.

The general LOD framework in Fig. 3 is not our contribution but to introduce the background task and how paired VL (vision-language) inputs predict target objects. The technological novelty of our paper is that we pioneer the agentic workflow by designing reasoning/planning, tool use, and reflection steps as a whole pipeline to improve VL alignment from a data-centric perspective. In addition, the states/actions are specifically designed based on our data analysis of visual objects and corresponding language expressions. The improved VL data continuously benefit all LOD methods even though their model architectures evolve.

The works seems to directly use the LLM and VLM. There is no real innovation for both LLM and VLM.

While we use the LLM and VLM as our tool models in the workflow, we would like to emphasize that our work is not a direct application of them, and the innovation resides not in the model itself but in the design of the proposed agentic workflow. Our design rationale formulates different models (LLM and VLM) as a whole system through the coherent workflow rather than an individual off-the-shelf integration. The following are three aspects of our innovation:

1. We pioneeringly develop a coherent agentic workflow containing planning, tool use, and reflection steps for VL data alignment. We carefully design each step to ensure the overall performance.
2. Our state/action design is based on our VL data analysis in Ln 355-364, where we first categorize hallucination types and develop actions accordingly. Fig. 6 and 7 show that our actions effectively refine raw expressions.
3. Our workflow pioneeringly improves data quality from the VL alignment perspective. Our data-centric design benefits all the LOD model training regardless of their model architecture evolvement.

Besides, we empirically find that adding global image captions (e.g., Fig. 5) further improves the reasoning ability in our planning step. This is because we find out the LLM reasons more effectively in pure language rather than VL form. Moreover, we utilize LoRA tuning (i.e., Ln 344) to maintain the original reasoning ability of LLM. As such, these efforts indicate that we intend to integrate different models more coherently through agent characteristics rather than directly using them.

Dear Reviewer e1U7,

We hope this message finds you well.

Thanks again for your insightful suggestions and comments. As the author-reviewer discussion time is limited, we want to ensure that we address any remaining uncertainties or questions you may have.

After carefully considering your comments, we have taken the necessary steps to provide additional clarifications, particularly how our workflow reduces hallucinations and the novelty of our method. We hope that our explanations and additional details contribute to a clearer understanding of our workflow pipeline.

We are open to providing further clarifications or conducting additional experiments if needed. Please feel free to reach out to us with any further inquiries.

Once again, we are thankful for your time and attention to our work.

Best regards,

The Authors

Dear Reviewer e1U7,

As the author-reviewer discussion time is one day left, we sincerely request a reply from you again.

We have carefully responded to your comments: 1. Regarding the question about novelty, which Reveiwer LXzJ, J3CB and M1hG have acknowledged, we have provided a detailed explanation to reemphasize the novel contribution of our method. We have also clarified that Fig. 3 is not our contribution but to introduce the background task. 2. Regarding the hallucination problem, we have further illustrated the three detailed schemes to solve it in our workflow.

We would like to ensure that the above responses have answered your uncertainties or questions. We are open to providing further clarifications or conducting additional experiments if needed. We sincerely look forward to your reply with any additional inquiries.

Once again, we genuinely appreciate your time and effort in our work.

Best regards,

The Authors

Dear Reviewer LXzJ,

We would like to thank you for providing valuable comments, and we answer the concerns raised below. We have included all discussion results in our revised manuscript (content in orange color).

Second paragraph of section 3.1: Since the authors are generating the initial training data for LOD (that they will later re-align with Real-LOD) and not using the data that is out there from previous work, the setup and how the data generation is accomplished should be clearly explained. The authors just mentioned the name of the model they use.

• For the initial expression generation with LLava, the authors should clearly state the prompts used, formatting structure, number of expressions generated, etc.
• The authors also mention, “vicuna is used to diversify the generated descriptions”. What does “diversify” mean in this context? How exactly is it done, given the generated data?

We appreciate the valuable suggestions and make the clarification here. Since the initial expression generation before re-aligning is not the main contribution of our paper, we simplify the presentation and provide an overview of the whole pipeline in the Sec. A of the appendix where are two steps to generate initial language expressions.

1. Regarding the details of expression generation through VLM, we use LLaVA to generate 673k expressions for 188k images with 336.5k objects, following previous works [1, 2]. For each object, we randomly select two prepared prompts (shown in Table 8 of Sec. G.1) with an image and corresponding category for LLaVA to generate expressions.
2. As for the second point, "diversify" means generating synonyms to expand the richness of language expressions. Vicuna is introduced to achieve this proposal by redescribing the given expression, and it expands the number from 673k to 1,346.1k. The prompt is shown in Table 8 of Sec. G.1. We repeat the process two times for each expression.

Revised Contents:

1. In the second paragraph of Sec. 3.1, we have modified the uncertain part and added the reference to Sec. A.
2. In the Sec. A, we have added a detailed explanation of the initial expression generation.
3. In the Table 8 of Sec. G.1, we have provided the details of the prompt template.
4. We replace the "initial" with "raw" to keep the consistency of the paper.

[1] Dang et al., Instructdet: Diversifying referring object detection with generalized instructions. ICLR, 2024.

[2] Pi et al., Ins-detclip: Aligning detection model to follow human-language instruction. ICLR, 2024.

Since the authors are fine-tuning “ChatGLM-6B” as their agent (which is responsible for choosing the state and action to be executed), it seems like the agent does not have access to the image, and it reasons only based on the textual prompts. However, the figures and texts give the impression that the image is also used as input to the agent to choose the state/action. This should be clearly stated in the text that the agent does not have access to the image.

• Also related, it seems like the agent does not use the bounding box location either. This should also be clarified.

We appreciate the valuable suggestions and make some clarification on the agent input here:

1. We agree that the proposed agent does not have access to the image, and we have clarified this point in Ln 274-277 to address the confusion. The reason why we do not provide the agent with the image is that we find LLM reasonings are more accurate in pure language form than in VL (vision-language) form.
2. The bbox locations are indeed not used for the reasoning process of agent, which we have clarified in Ln 276. However, these locations are provided for specific tools in the tool use step. They help LLM avoid positional errors in rewriting and enable VLM to focus on the object region during re-perception.

Revised Contents:

1. We have clarified the above presentation to address the confusion of agent input in Sec. 3.2.
2. We have also provided more detailed information about the agent input in Sec. 3.2.

Line 260: Vague statement: “When using these models, Real-LOD agent adaptively modulates visual content and text prompts based on its reasoning thoughts”. It is not clear which aspect of the agent this statement is referring to.

We apologize for the unclear statement. This statement refers to the capability of the agent to adaptively decide the visual content and text prompt for the chosen tool, i.e., the "Image editing" and "Prompt" parameters, as shown in Fig. 5. We have modified this statement in Ln 302 (i.e., Ln 260 in origin manuscript): "Based on its reasoning about the state of the current expression, Real-LOD$^{agent}$ adaptively modulates visual content and text prompts by setting up "Prompt" and "Image editing" parameters for scheduled tools. Then, the tool can be used effectively to get desired responses from VLM to improve the expression refinement.".

Further to the previous point, the agentic pipeline as is now has quite a lot of components. It might be very beneficial if any of them can be removed without significant loss in performance.

We appreciate the valuable suggestion. We argue that there are not quite a lot of components in the proposed workflow, and each component is significant because our design principle considers all the components as a whole pipeline. In addition, the computational cost of our workflow is affordable and performs off-line without affecting the LOD model inference, as shown in Sec. 4.3.

Besides, we conduct a more detailed ablation study of our agentic workflow. The evaluation results of the success rate are as follows. The details of the experimental setup can be found in Sec. 3.3. In the following table, the "w/o Planning" is the same as the random selection schema in Sec. 3.3. "w/o Cyclic Workflow" indicates the workflow with only one cycle. The results intuitively illustrate the importance of each component to our agentic workflow.

We have concluded the following table in Sec. J of the appendix. This question will also motivate us to explore building a more lightweight workflow in the future.

The paper several times mentions the model trained is a "prevalent" model -- can this be backed up with other works that use the same model? Comparing to these more explicitly would make the contribution of this work stronger

We appreciate the valuable suggestion. The prevalent model can be backed up with other works. However, to our knowledge, no existing methods directly related to ours improve the VL (vision-language) alignment ability with agentic workflow from a data-centric perspective. We have considered this problem and performed various experiments in Sec. 4 and Sec. E to demonstrate the effectiveness of our workflow. Especially in the ablation study of Sec. 4, we conduct three data configurations (i.e., A, B, and C forms) to train the "prevalent" model. In A form, all the generated data pairs are used for training, which can serve as the application of previous work[1]. This work aims to diversify the object expressions. The 3% improvement of AP-des in OmniLabel-COCO brought by C forms, where we use the re-aligned data pairs via our workflow for training, is strong evidence to demonstrate the effectiveness.

[1] Dang, et al. InstructDET: Diversifying Referring Object Detection with Generalized Instructions. ICLR 2024.

The training of the agent is not properly explained. Some things that should be there are 1) how many samples were used to train the agent? It says that these were manually created by the authors using the outputs of the various tools. Did this include all the recurrent steps?

We appreciate the valuable suggestions and make some clarification on agent fine-tuning here:

1. We use 15k samples to train the agent, as shown in the first paragraph of Sec. 3.3.
2. The training data do not include all the recurrent steps. This is because the planning process of the agent in each recurrent step is the same and we only need to focus on the behavior of agent in one recurrent step.

We have provided more details about agent fine-tuning in the first paragraph of Sec. 3.3.

Minor: Section 4.1 introduces forms A,B,C which are then not references in the table and is a bit confusing

Thanks for pointing out this problem. We have added references to the A, B, and C forms in Table 4 (i.e., origin Table 1).

Dear Reviewer J3CB,

We would like to thank you for providing valuable comments, and we answer the concerns raised below. We have included all discussion results in our revised manuscript (content in orange color).

The agent has access to a variety of actions to choose from (from the states it can be) and there is some intuition that the 6 mis-alignment reasons (category, attribute, etc) will each be beneficial for some of these failure modes. It would be good to see if the used actions and tools actually reflects that intuition with some analysis on which actions were chosen when different mistakes were made

We appreciate the valuable suggestion. In our paper, we have provided several examples to analyze which states/actions were chosen when different misalignments may happen. The following are some of them (omit state 1 and state 2):

1. When Real-LOD is in state 3, uncertain about the category or attribute of the target object, it plans to execute action 3. This action involves the object crop operation, which directly crops the target object for the VLM, and then VLM can better re-percept focusing on the target without introducing other noise. In addition, the question prompt will be redesigned so that VLM can focus on the target object itself.
   • Fig. 14 in the appendix shows an example of the first cycle. The expression describes the attribute (i.e., color) of the 'dining table.' For examination of correctness, Real-LOD crops the table and redesigns a specific question prompt that 'Questions:1.What is the color of the table?2. Is the table made of wood or another material?' for VLM. Then, Real-LOD can acquire information that '1. The color of the table is brown.2. The table is made of wood.'
2. When Real-LOD is in state 4, uncertain about how the target object relates to its surroundings or accessory objects, it plans to execute action 4. This action involves the extended object crop operation, which provides a more extensive region covering the target objects with surroundings for the VLM, and then VLM can better re-percept rather than a simple object crop or a whole image. In addition, the question prompt will be redesigned so that VLM can focus on the relationship between the target object and its surroundings.
   • Fig. 5 shows an example in the first cycle. The expression describes the horse and its relation to the surroundings. For examination of correctness, Real-LOD crops a larger region of horse and redesigns a specific question prompt that 'Questions:1. Is the horse walking on sand or on a dirt path?' for VLM. Then, Real-LOD can acquire information that 'the horse is walking on a dirt path, not on sand.'
3. When Real-LOD is in state 5, uncertain about how the target object relates to the whole image, it plans to execute action 5. This action involves highlighting the object region with a red rectangle, and then VLM can better re-percept the location or relationship/interaction with the further or larger objects in the whole image. In addition, the question prompt will be redesigned so that VLM can focus on the relationship of the target object further.
   • Fig. 16 shows an example in the second cycle. The expression describes the bus and its relation to the whole image (i.e., location). For examination of correctness, Real-LOD highlight the bus without cropping the image and redesigns a specific question prompt that 'Questions: 1.Is the bus on the right side of the image?' for VLM. Then, Real-LOD can acquire information that 'Yes, the bus is on the right side of the image.'

As demonstrated in Fig. 7, Real-LOD achieves high success rate of expression refinement in 6 aspects, especially: Relation (relation to surrounding objects, such as apple on table), Accessory (relation to accessory objects, such as person with a hat), Location (relation to whole image, such as bus on the right side of image) and Behavior (relation to further objects, such as man playing frisbee while a group of people watch).