Expert-level protocol translation for self-driving labs

Below, we present several real-world examples to illustrate these distinctions. In our implementation of the pipeline, we employ a state-of-the-art dependency parser [1] alongside a state-of-the-art LLM-based NER model [2].

References:

[1] Honnibal, M. and Johnson, M. (2015). An improved non-monotonic transition system for dependency parsing. In Annual Conference on Empirical Methods in Natural Language Processing.

[2] Xie, T., Li, Q., Zhang, Y., Liu, Z., and Wang, H. (2024). Self-improving for zero-shot named entity recognition with large language models. arXiv preprint arXiv:2311.08921.

Here, we provide a realistic example of XDL machine-executable code used to operate the devices in a self-driving laboratory for Chemistry [1].

Original protocol: 

2,2'-dinitro-6,6'-dimethylbiphenyl (39 g, 0.14 mol) was dissolved in 100 ml ethyl acetate in a hydrogenation vessel. Palladium on Carbon (10%, 5.5 g) was added. The system was evacuated and H2 added to a pressure of 28 psi. The reaction was left until no further uptake of H2 could be detected. The solution was filtered through celite and the solvent evaporated to give the product diamine in 100% yield.


Machine executable code:

<?xdl version="1.0.0" ?>
<XDL>

<Synthesis>

  <Hardware>
    <Component
      id="cartridge_celite"
      type="cartridge"
      chemical="celite" />
    <Component
      id="reactor"
      type="reactor" />
    <Component
      id="rotavap"
      type="rotavap" />
  </Hardware>

  <Reagents>
    <Reagent
      name="2,2'-dinitro-6,6'-dimethylbiphenyl"
      id="2,2'-dinitro-6,6'-dimethylbiphenyl"
      role="reagent" />
    <Reagent
      name="H2"
      id="H2"
      role="reagent" />
    <Reagent
      name="ethyl acetate"
      id="ethyl acetate"
      role="reagent" />
    <Reagent
      name="palladium on Carbon (10 %)"
      id="palladium on Carbon (10 %)"
      role="reagent" />
  </Reagents>

  <Procedure>
    <AddSolid
      vessel="reactor"
      reagent="2,2'-dinitro-6,6'-dimethylbiphenyl"
      mass="39 g" />
    <Dissolve
      vessel="reactor"
      solvent="ethyl acetate"
      volume="100 mL"
      temp="25 °C" />
    <AddSolid
      vessel="reactor"
      reagent="palladium on Carbon (10 %)"
      mass="5.5 g"
      stir="True" />
    <EvacuateAndRefill
      vessel="reactor" />
    <Add
      vessel="reactor"
      reagent="H2"
      volume="0"
      stir="True"
      speed="40.0" />
    <FilterThrough
      from_vessel="reactor"
      to_vessel="rotavap"
      through="celite" />
    <Evaporate
      vessel="rotavap"
      time="30 min" />
  </Procedure>

</Synthesis>

</XDL>

References:

[1] S. Hessam M. Mehr et al., A universal system for digitization and automatic execution of the chemical synthesis literature. Science 370, 101-108 (2020).

The proposed solution is a portfolio of standard applications of existing tools or well-known algorithms, which is less interesting and novel from a machine learning perspective.

Thanks for the comment. In this work, we study the problem of translating experimental protocols designed for human experimenters into formats suitable for machine execution. Our primary motivation is to bridge the existing gap between machine learning algorithms in the field of AI for science, such as molecular design, and the grounded experimental verification facilitated by self-driving laboratories. We appreciate the reviewer's recognition that "the required background knowledge is non-trivial". Indeed, conventional workflows for setting up self-driving laboratories and conducting physical experiments necessitate deep integration with domain experts, significantly impeding the progress of machine learning researchers in verifying and iterating their findings. Consequently, our framework aims to provide an infrastructure that enables these researchers to advance their machine learning algorithms and seamlessly validate their findings, thereby closing the loop of automatic scientific discovery.

To meet the requirements of such infrastructure, we conduct a systematic study to identify existing gaps in protocol translation between human experimenters and automatic translators in self-driving lab. Accordingly, we develop the three-stage framework that integrates cognitive insights from human experts with approaches from program synthesis, automata construction, and counterfactual analysis. At the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations, thereby making explicit the operation-condition mappings and the control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, addressing the latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in the protocol dependence graph. This graph conducts counterfactual reasoning to detect potential conflicts or shortages of execution resources and to identify inappropriate combinations of operations in execution sequences.

The targeted DSL is relatively simple, and the proposed solution consists of ad-hoc design choices (particularly spatial-temporal dynamics), which may not generalize well to DSLs with richer features

Thanks for the comment. In this study, we aim to investigate the development of automatic translators for executing experimental protocols in self-driving labs. To execute the instructions on Internet-of-Things-connected hardware devices, such as valves, pumps, and reactors, protocols must ultimately be formatted in JSON-style configuration files, which represent a mainstream format in hardware-software communication (see C.3-1).

Although DSLs with syntactic and semantic features differing from those of JSON-style DSLs for self-driving labs are beyond the scope of this paper, generalizing the framework to DSLs with other language features represents a significant direction for future research. We appreciate the reviewer's suggestion in this regard.

We acknowledge that the core components of our framework are designed in an ad-hoc manner. The design guidelines are derived from both a systematic study of the required cognitive capabilities in protocol translation and established computer science theories. Rather than directly devising engineering solutions tailored to the specific problem, we decompose the problem to abstract subproblems, including symbolic regression, flow analysis, and counterfactual reasoning. We consider these subproblems as scientific challenges and develop solutions from the conceptual level to the implementation level in a top-down approach.

Specifically, the scientific challenge behind the design of spatial-temporal dynamics lies in the extremely long-tail distribution of historical run-time error cases. To address this issue, we propose leveraging foresight simulation by contextualizing individual operations within both the spatial dimension, i.e., the specific assignment of resources according to capacity requirements, and the temporal dimension, i.e., the specific precondition and postcondition of resources according to their properties. The division of spatial and temporal dimensions is mutually exclusive and echoes the two major aspects of computer programs --- computation resources and control logic. Thus, although ad-hoc, the design of spatial-temporal dynamics targets the underlying scientific problem behind the superficial challenges and holds the potential for generalization to other DSLs. Extending the application scope of our framework is a significant direction for future work.

What off-the-shelf tools are used in the pre-processing to extract actions and entities? Are they LLMs? Are there important differences between the extraction of action/entities and the extraction of reagent entities?

Thanks for the question. We employ the SpaCy Dependency Parser to analyze the syntactic structure of protocols, which allows for the extraction of verbs and the identification of associated objects and modifiers. After parsing, these verbs are aligned with corresponding operational actions in our DSL by maximizing the cosine similarity between their word2vec representations and those of the DSL operations. Furthermore, we utilize a few-shot model based on LLM to accurately identify and classify entities within the text. The rationale of integrating LLMs with classical parsing techniques lies in leveraging the advanced natural language processing capabilities of LLMs while mitigating their inherent uncertainties.

There are significant differences exist between various stages of this pipeline (see C.3-2).

This series of examples demonstrates how our system tracks the required capacities at each step of the protocol by contextualizing the step into the spatial dimension.

This series of examples illustrates how our system tracks the preconditions and postconditions at each step of the protocol by contextualizing the step into the temporal dimension.

The example presents as follows — the completion of two types of parameters at the semantic level is included: for instance, determining the configuration parameter for an operation, where human experts rely on personal experimental experience; and inferring the required reagents for one step, where human experts use contextual reasoning. When the context is not sufficiently clear, human experts cannot infer the known unknowns within a single sentence.

This series of examples demonstrates the superior performance of our system at the syntax level when processing relatively short sentences.

This series of examples illustrates the challenges faced with longer sentences due to the diversity of actions and the multiple parameters.

Tab. 2-1-5: Execution level - Capacity of resources

Tab. 2-1-6: Execution level - Safety of operations

Tab. 2-1-4: Semantic level - Latent semantics of unknown unknowns

Tab. 2-1-2: Syntax level - Operation control flows

Tab. 2-1-3: Semantic level - Latent semantics of known unknowns

Tab. 2-1-1: Syntax level - Operation-condition mapping

What specific optimizations could be applied to reduce the computational requirements of the proposed framework?

Thanks for the question. Computational efficiency is always a topic of interest when evaluating new computational frameworks. Let us consider a new coming protocol with $k$ steps, with each step configured by a constant number of parameters, denoted as $\epsilon$. At the syntax level, the primary computation bottleneck arises during DSL program synthesis, where the EM Algorithm exhibits a worst-case complexity of $O(\epsilon^k)$. This is a highly conservative estimate, as mainstream optimization approaches can solve the EM much more efficiently. At the semantics level, the bottleneck occurs during reagent flow analysis, which consumes $O(k^2)$ complexity. Notably, only approximately 10% of the steps are included in the nested loop for reagent flow construction, as about 90% of the steps are linearly connected. At the execution level, the protocol execution model also exhibits $O(k^2)$ complexity, encompassing both forward and backward tracing. This can be optimized by replacing the full tracing strategy with a sliding window built upon the topological dependencies between steps.

Although the complexities of the algorithms at these three levels are tractable, there is substantial room for improving the efficiency of the framework. Investigating methods to speed up the translation process for protocols with extremely high complexity would be a valuable area of research. We are committed to making the computational framework as accessible as possible for all research teams.

In addition, our proposed framework functions as an auxiliary module for LLMs, supporting the use of off-the-shelf LLMs such as GPT and Llama without the need for domain-specific fine-tuning. Costs associated with calling commercial LLM APIs are quite affordable. We selected OpenAI's gpt-3.5-turbo-0125 model for our experiments. Across 75 test protocols, we executed 1816 queries to achieve syntax-level translation, resulting in structured protocols. At the semantic level, we conducted 4062 queries for completion tasks (including translating protocols retrieved from training dataset). Consequently, our expenditures were approximately 17 USD in total.

How does the system ensure the safety and correctness of translated protocols, especially in high-stakes domains such as medical and clinical research?

Thanks for the question. In general, ensuring the safety and correctness of translated protocols in high-stakes domains is an exceptionally challenging task. Several factors contribute to these challenges, including accurately mapping operations to their corresponding configuration parameters, precisely parsing control flows from natural language, completing latent semantics with domain-specific knowledge, inferring missing or omitted key information, tracking resource capacities, and verifying the safety of run-time execution of experiments. Even minor errors in these areas can significantly compromise the safety and correctness of translated protocols. Consequently, we have made specific efforts in response to these challenges.

At the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations. This approach explicitizes operation-condition mappings and control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, thereby addressing latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in a protocol dependence graph. This graph facilitates counterfactual reasoning to identify potential conflicts or shortages of execution resources and inappropriate combinations of operations within execution sequences.

We provide several illustrative examples to demonstrate these concepts (see C.2-1).

The paper could benefit from a more detailed analysis of the types of errors made by the automated translator compared to human experts, which would help understand the limitations and areas for improvement.

Thanks for the suggestion. Here we present a detailed analysis of the errors made by our proposed automatic translator compared to human experts. We discuss the potential improvements of the translator accordingly.

At the syntax level, the major difference being in the analysis of long sentences in natural language. Human experts analyze the parameters of events/actions or multiple actions in long sentences with ease, while for our approach, there are sometimes problems with the correspondence between action and parameter (see C.2-2-1).

At the semantic level, when supplementing known unknowns, human experts tend to infer parameters based on established protocols outside their expertise; when supplementing unknown unknowns, human experts tend to transfer their knowledge from familiar domains to protocols in various fields. Our system, however, completes parameters based on all collected protocols, which is essentially the opposite of the transfer process used by human experts (see C.2-2-2).

At the execution level, human experts track capacity primarily based on prior knowledge, subsequently using context to judge the appropriateness of the equipment used. In contrast, the machine extracts the entire flow process, enabling it to calculate each step and ensure that the capacity tracking is scientifically sound and reasonable (see C.2-2-3).

Can the authors provide more details on how the system handles ambiguous or incomplete protocol instructions that may be common in real-world scenarios?

Thanks for the question. Here, we present a series of case studies to elucidate the the specific behaviors of components within the proposed three-stage framework at the syntax, semantics, and execution levels (see C.2-3).

Here we provide a series of case studies to illustrate the distinctions between the behaviors of the components within our proposed three-stage framework and those of the baselines qualitatively.

Imagine a self-driving kitchen that automatically prepares all ingredients and executes all procedures for cooking a meal according to natural-language-based recipes. Such self-driving kitchens would also benifit significantly from translating human-oriented recipes into formats suitable for machine execution. In the following, we present a running example of such a translation, adapted from [1].

The protocol after pre-processing is as follows.

Pasta Bolognese

Yield: 2 plates

Ingredients:

- 8 [ounces] white fresh {pasta}

- 1 [floz] olive {oil}

- 1/4 [ounce] {garlic}; minced

- 4 [ounces] {onions}; chopped

- 4 [ounces] shallow fried {beef}; minced

- 1 - 1 1/2 [ounce] lean prepared {bacon}

- 1/3 [cup] red {wine}

- 150 [gram] raw {carrots}; thinly sliced

- 2/3 [ounce] concentrated {tomato puree}

- 4 [ounces] red {sweet pepper}; cut julienne

- 1 [ounce] {parmesan} cheese

Instructions:

Add the @oil@ to a large saucepan, heat to <300 F>, and saute the @onions@. 

After |2 minutes|, add the @garlic@. Keep on medium to high heat, and don't stir. 

After |2 minutes| more, add the @beef@.

Fry the @bacon@ in a separate pan, on high heat. Remove liquified fat when done.

Boil @pasta@ in a medium pan, until al dente (~|8 minutes|). Drain when done.

Once the @beef@ is done, add the @carrots@, @sweet pepper@ and @tomato puree@. 

Slowly add the @wine@ as well, to not lower the temperature. Let it simmer (but not boil) for |5-10 minutes|. 

Given the protocol as the input of our framework, the resulting DSL program is as follows.

add(slot = "oil", target = "large saucepan", container = plate_1, emit = mixture_1);

heat(target = mixture_1, temperature = 300F, container = plate_1, postcon = stop());

saute(target = "onions", container = plate_2, duration = 2mins);

add(slot = "garlic", target = mixture_1, container = plate_1, emit = mixture_2);

heat(target = mixture_2, temperature = 325F, container = plate_1, duration = 2mins);

add(slot = "beef", target = mixture_2, container = plate_1, emit = mixture_3);

heat(target = mixture_2, temperature = 325F, container = plate_1, postcond = check_done(target = "beef"));

fry(target = "bacon", temperature = 350F, container = pan_1, postcond = remove(target = "liquified fat"));

boil(target = "pasta", temperature = 212F, container = pan_2, duration = 8mins, postcond = drain());

add(precond = check_done(target = "beef"), slot = ["carrots", "sweet pepper", "tomato puree"], target = mixture_3, container = plate_1, emit = mixture_4);

add(slot = "wine", target = mixture_4, container = plate_1, pace = 1mL/s);

simmer(target = mixture_4, temperature = 211F, duration = 7.5mins);

In this example, we observe that the natural-language-based recipe possesses ambiguities and omissions. Our translation framework addresses these challenges by structuring the recipe at the syntax level, completing the latent information at the semantics level, and linking the programs with necessary resources, such as the usage of plates, at the execution level.

References:

[1] Roorda, Auke. "Corel: A DSL for Cooking Recipes." Diss. 2021.

Why is BLEU/ROUGE used as a metric? Can a more semantically aligned mode of comparison be used here.

This is a very good question. The same concern was considered during the development of our evaluation methodology. Direct comparisons across entire sentences under BLEU/ROUGE scores would indeed pose a problem as the reviewer mentioned. Therefore, to circumvent this issue, we convert all results into a standardized JSON-style format for data representation, and comparisons are made between key-value pairs rather than entire sentences, effectively resolving the metric concern.

Let us consider the example mentioned by the reviewer: we represent the two sentences "... pour hot water ..." and "... pour cold water ..." in the following JSON-style format.

{
 op: "pour",
 reg: "water",
 T: "hot",
}

{
 op: "pour",
 reg: "water",
 T: "cold",
}

The comparison between the two sentences is then transformed into a comparison between two JSON code blocks. We calculate the similarity score cumulatively based on the similarity between the values of matched pairs of keys. For instance, for the key "temperature", the values "hot" and "cold" yield a low similarity score under the ROUGH, BLEU, and even the Exact Match metrics. As "temperature" is one of the major keys within configuration parameters, a high penalty in this dimension significantly affects the cumulative similarity score. With this fine-grained comparison metric, we can comprehensively track the distinctions and commonalities between results without losing expressivity regarding the quantities.

We also acknowledge that there are advanced evaluation metrics, especially in the recent works where LLMs are leveraged as external judges and achieve considerable performance in general testing cases. Our choice of "less advanced" metrics is driven by the intention to focus specifically on domain-specific knowledge, which constitutes the primary scope of this paper and may be relatively sparse in general LLMs. Nonetheless, the exploration of more sophisticated evaluation metrics represents a promising avenue for future research, and we appreciate the reviewer's perceptive recommendation in this regard.

Since the tool aims to save time for domain experts, it should be demonstrated how much it is easier to write down instructions in this format as compared to the machine usable one directly.

Thanks for the comment. The scope of our proposed framework, in the current stage, is to automatically translate human-oriented protocols into formats suitable for machine execution, rather than helping domain experts creating new ones in an easier format. The goal is to transfer knowledge in a conventional lab into a format suitable for self-driving labs. Therefore, the human-oriented protocols used in this translation are existing protocols previously designed for human operators in the conventional labs, thus coming with no extra cost.

In contrast to conventional protocol translation processes, which require domain experts to manually develop rules and functions based on specialized knowledge, our proposed automatic translator attempts to eliminate the need for such expert intervention. Domain experts are not involved in either the development or the execution stages of our translator. Therefore, the evaluations in our paper mainly focus on translation performance rather than generation efficiency. We appreciate the reviewer's insightful suggestion and will make revisions accordingly for better clarification.

While this demonstrates the system's usefulness in translating such inputs, it is unclear how well it may generalise to inputs following looser terminology / formatting.

Thanks for the comment. The general applicability of our proposed framework beyond experimental sciences can indeed be a common concern. The core value of translating natural-language-based protocols into formats suitable for machine execution substantially lies in facilitating experiments in self-driving labs, thereby accelerating scientific discovery. Experimental protocols come with unique properties and challenges, such as the fine-grained incorporation of domain-specific knowledge, the non-trivial dependency topology between operations, the long-horizon lifecycles of intermediate productions, and the necessity for precise execution without run-time errors. These factors shape the scope of our research problem, emphasising the need to handle protocols with stringent terminology and formatting.

Despite the specific scope of this paper, we are open to exploring the potential for generalizing our framework to other domains with similar challenges as those found in scientific experiments, such as cooking (see C.1-1).

Qualitative, how does the approach behave differently to the baselines? What does each component of the system contribute?

Thanks for the question. The rationales for the components within our proposed framework are grounded in both empirical and theoretical considerations. We develop the three-stage framework that integrates cognitive insights from human experts with approaches from program synthesis, automata construction, and counterfactual analysis. At the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations, thereby making explicit the operation-condition mappings and the control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, addressing the latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in the protocol dependence graph. This graph conducts counterfactual reasoning to detect potential conflicts or shortages of execution resources and to identify inappropriate combinations of operations in execution sequences (see C.1-2).