Agent-DocEdit: Language-Instructed LLM Agent for Content-Rich Document Editing

We thank Reviewer gr6V for the constructive feedback and acknowledging that our proposed framework is useful for real-world challenges. We address the comments below.

[W0] System breakdown analysis.

The suggested aspects are below:

Avg lines of code: 10.75 with 1.4 main API calls.

The syntax error rate is ~33.3% as interpreted by Table 1.

The semantic error rate (number of errors/edits) within our human studies subset is ~34.6% without feedback and ~12.3% with feedback, where the feedback provider reaches 71% accuracy on a held-out test set.

[W0] Limitations and paper clarity.

We apologize for the potential confusion on the software dependent claims. Our APIs are indeed designed to be generally applicable and software-agnostic. The dependency claim is made to clarify that those APIs are not exactly the final interfaces for manipulating the underneath document structure/contents, where some efforts would be made to map them to the actual execution functions (e.g., mapping from our Python APIs to corresponding design editing software code.) We will supplement a list of supported actions/edits to each of our API as suggested (e.g., modification supports curve, italic, bold, textual edits, etc.).

”Grounding” means localizing an entity (phrase) in images.

[W1] Programmatic limitations of the system.

Yes. For the zebras example, the grounding module will propose boxes grounded to the word “zebra”, and the edit program will loop over them with applying corresponding actions. Our framework can process multiple contents by compiling a procedure invoking multiple API calls (with proper logic) to the contents. However, requests may need to be more specific, e.g., the user needs elaborate the meaning of ”daytime” scene (e.g., ”change moon to sun”) as inferring implicit editing is left as a future exciting work.

[W2] System runtime.

In practice, the trained feedback provider is launched in a “serving mode” on a separate GPU queuing for inputs, and if all other model weights are in-memory, the overall inference takes roughly 10-15 seconds per each request. The editing software may add another 10-15 seconds for content re-rendering.

[W3] Assumption of document structure.

We acknowledge such a limitation with an emphasis on presumption of existing editing software here. We foresee a combination of pixel-generative models and our framework as an exciting future direction for achieving even more general use cases.

We are thankful to Reviewer 4fVM for the very positive feedback, where our proposed framework is very convincing in the realm of content-rich document editing. We address the comments below.

[W0] Comparison to human executions.

We would like to point out that the ground truth actions/components from the DocEdit dataset reflect what the annotators believed would be the actual actions for the edit requests. As a result, the actions performed that our framework is trained to mimic is "highly aligned" with the actual human execution.

The DocEdit dataset itself also features the modified (edited outcome) documents, representing the results of the human executions.

As for the format of the feedback, we make the sentences short and concise so our framework can perform more deterministic revisions, and since they are deterministic, we do believe that human editors would take similar actions upon receiving the same feedback. However, we also think it is an interesting extension of ours to cope with (and train) more complex feedback (provider), where some learnable modules will be integrated to comprehend the more unrestricted natural language feedback.

[Q1] Paper refinements and section rearrangements.

Thank you for the suggestion on restructuring our paper. We will follow it when revising the final manuscript.

[Q2] The phrase regarding “linguistics”.

The ”linguistic request” here refers to the “textual (or verbal) request” as some editing requests could be non-verbal (such as dragging the contents directly). We apologize for the misuse of the particular terms and will revise them according to your suggestions.

Thanks for the response!

We thank Reviewer 5Bmb for all suggestions and constructive feedback, where we address them below.

[W1,2] Details of feedback mechanism.

Negative samples are constructed by manipulating the contents with exactly opposite or wrong actions (we know right/wrong from the ground truths) against a sampled request – guaranteeing the resulting samples contradict it. Positive/negative samples have equal amounts. We tried paraphrasing the requests with LLMs, however, it did not bring significant benefits. The feedback module reaches 71% accuracy on a held-out test set.

[W3,Q1,3] Analysis of actions/components.

The error rates (w/o feedback) of top 3 frequent actions in the human subset:

REPLACE: 22%

MODIFY: 30%

MOVE: 37%

Our system does particularly well at REPLACE. We hypothesize MOVE can benefit from improved target position predictors. Our grounder performs quite evenly well on all kinds of components – no particular bias against or towards kinds of components is observed.

For new tools/actions, as long as comprehensive descriptions/examples are provided, our system can generate effective plans. (The grounder and feedback module are agnostic to these.) Clearer and more comprehensive guideline prompts give higher quality in the edit programs.

[W4] Details of human studies.

We recruited 5 college students with linguistic/CS majors and presented them with the edit request alongside outcomes from all 4 methods (scrambled). They ranked them (1-4) on how aligned the samples were to the request. Each instance is ranked by all evaluators, and averaged as in Table 4.

[W5] Limitations of feedback mechanism.

There are rare cases even semantic/syntactic verifications could overlook, such as complete style changing. Such kinds of requests are not currently dealt with by our system, however, our metrics and human judgements would be able to spot these and score them appropriately. The ”wrongly grounded component” in Sec.2.4.1 is exactly designed to handle when the system accidentally changes non-indicated components.

[W6] System robustness.

The feedback module is proposed exactly for this reason: to make the system capable of recovering from errors. Once the edit program is syntax-bug-free, the semantic errors may come from failed grounding, and the feedback will help search other proposed boxes for more correct components to act on. Component robustness can be inferred from Table 1 (code pass rate and parameter accuracies), and Table 2.

We are thankful to Reviewer Xzje for the comments and that the reviewer thinks our work is novel and the exploration of agents in complex tasks is vital. We address the comments below.

[W1] Additional discussions on the system design choices.

Thanks for the suggestion. Below we discuss additional design choices which will be included in the final manuscript.

Program Generator: The edit program generator is built on top of instruction tuned code-gen. models. Our early experiments show that OpenAI’s GPT is better at attending to long and complex API lists (via system prompt), than other concurrent models such as Code-Llama. However, our framework does not depend on specific code-generators and any of them can be used by our framework.

Grounding Modules: In addition to the GLIP used in this work, there are a few other decent grounding models such as Grounding-DINO [1]. We chose GLIP for its simplicity and high adaptability to other data domains. We further improve the phrase extraction with the method in Sec 2.2.

Feedback Provider: The feedback for the editing outcomes requires examination of both the input document contents (images) and the edit requests, and the actual feedback is a generated sentence regarding the success/failure of the editing. Consequently, strong multimodal LLMs, such as LLaVa, naturally emerged as the most suitable candidate for this task.

[1] Liu, Shilong, et al. "Grounding dino: Marrying dino with grounded pre-training for open-set object detection." arXiv 2023.

[W2] Success and failure cases.

In the final manuscript we will discuss success/failure cases. Our general observations are: The two (instruction-tuned) generative baselines often exhibit severe degeneration of the parts that are supposed to be intact, or fail to capture the intentions of the user editing request. Our modularized agent framework more successfully interprets the user requests and manipulates the components much more accurately. The feedback mechanism alleviates initial errors from both the edit program generation as well as inaccurate grounding.

We suggest that, for instructed document editing without a ready-to-use editing software (as in our work), an interesting future work could be an integration of our modularized framework into the instruction-tuned generative pipeline, making it more robust and faithful to the input requests.