SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems

Dear reviewer Gxzq,

It looks like your question Q2 got cut off - "Q2) In Figure 1, the key-value template has only a colon to separate the key and the value. Have you tried adding a space in between? I expect...". Can you provide the rest of this question?

Thank you!

Dear Reviewer Gxzq,

Thank you for your review!

• [W1: LightGBM]: We are using the gradient-boosted decision tree (gbdt) model. We used 1000 estimators. We conducted a grid search over the learning rate, number of leaves, subsampling rate, feature fraction, and min number of data in leaf parameters. The ranges and best values selected based on this hyperparameter sweep are provided in Tables 6 and 7 in the Appendix. We will improve our presentation here in the main text in an updated version of the PDF.

• [W2: LLM-generated captions vs. templates]: Our work considers the use of both a) templated system captions (the colon-separated attribute names and values) and b) “conversational”-style natural language captions generated by an LLM. Some advantages of (b) include: the ability to use the LLM to augment the training data via prompt augmentation (see Section 6.5), the ability to leverage the LLM to invent descriptions for you when trying to come up with a description of a complex system or data (see [1] for a recent example trying to describe a smoke buoyancy simulation), and the potential to use the LLM to rephrase attribute names or values that may appear inscrutable to a non-expert (see our response [W5 - non-experts] to Reviewer 2gzl about this). We highlight that unconstrained, free-form text captions have been used extensively for text-audio and text-music multimodal modeling [2]. This is useful when, for example, a user wishes to search for a song but can only vaguely describe it. We draw an analogy to the application of complex system design—in early stages, an engineer may only have a vague idea about the characteristics that the final system will have. However, as mentioned below [W3], we leave conducting interactive user studies for future work.

• [W3 - user evals]: The reviewer is correct that we did not conduct a study to understand how non-expert users perceive our method. Our empirical evaluation is guided by the question of whether templated captions and LLM-generated captions can achieve good regression performance on real-world systems. The focus of this paper is thus on first establishing the technical feasibility of this general approach. We argue that previous work does not provide a conclusive answer to this question, and that our work confirms its feasibility. We will add a discussion in the revised PDF about the importance of conducting user studies to quantify how non-experts perceive language-augmented surrogate models.

• [Q1]: See response W2.

• [Q2 - tokenization]: We did not try adding a space instead of a colon in-between the key and value. We verified that the bert-base-uncased tokenizer has no issue tokenizing the colon separately. We provide an example at the bottom of the comment.

• [Q3 - numeric vars]: The building and wind farm simulators have numeric attributes that are “bucketed” by design, i.e., they only take on a fixed number of values. For example, the building square footage attribute only takes on 10 different values. For both datasets, the one-hot encoding baselines all use sklearn’s OneHotEncoder to create one-hot vectors out of all attributes.

• [Q4 - short vs. long]: We expected “long” captions to have larger error because the BERT text encoder is fine-tuned on captions of “medium” length, and Transformers are known to have difficulty with generalizing to longer sequences than seen during training. We could potentially improve generalization from “medium” to “long” captions by, for example, using a text encoder with a more advanced position encoding strategy that is less sensitive to the input sequence length, but we leave this exploration for future work. We will add more qualitative examples of short, medium, and long building captions to the appendix to help convey these intuitions better.

References:

[1] Zhou, Anthony, et al. "Text2PDE: Latent Diffusion Models for Accessible Physics Simulation." arXiv preprint arXiv:2410.01153 (2024).
[2] Huang, Qingqing, et al. "Mulan: A joint embedding of music audio and natural language." arXiv preprint arXiv:2208.12415 (2022).

Example:

>>> x = np.load('10_cap_ids.npy')
>>> tokenizer.decode(x)
'[CLS] building _ subtype : none | building _ type : smalloffice | number _ of _ stories : 1. 0 | sqft : 7500. 0 | hvac _ system _ type : psz - ac with no heat | weekday _ operating _ hours : 9. 25 | weekday _ opening _ time : 8. 5 | weekend _ operating _ hours : 8. 5 | weekend _ opening _ time : 6. 75 | tstat _ clg _ delta _ f : 0. 0 | tstat _ clg _ sp _ f : 75. 0 | tstat _ htg _ delta _ f : 6. 0 | tstat _ htg _ sp _ f : 68. 0 [SEP]'

Dear Reviewer 2gzl,

Thank you for your review and for praising the presentation, motivation, and empirical validation of the work.

• [W1,W2 - pretrained vs. finetuned embeddings]: We will clarify in our Introduction that we propose to fine-tune text embedding models initialized from pretrained weights. We are working on adding a new ablation to the PDF that shows the performance of our SysCaps-nl model without fine-tuning the BERT text encoder (i.e., only using the pretrained embeddings). We are also going to run an ablation without a "SOTA embedding model"---“SimCSE-RoBERTa-large” , which is the current most-downloaded text encoder on HuggingFace as of 11/19/24.

• [W3 - classifier hyperparams]: We will add the hyperparameter details for the multiclass classifier to the appendix. We implement the classifier on top of the text encoder by adding a linear layer for each attribute type, where this layer predicts logits for each attribute's classes. We use AdamW with lr=3e-4, early stopping with patience 5, batch size 128, and max epochs 100. We do not freeze the text encoder weights.

• [W4 - kv vs. nl]: On the comparison between SysCaps-kv on the building dataset and SysCaps-nl on the wind dataset, we believe the relative performance difference is easily explained. The wind dataset, which only has 500 total systems (300 training, 100 validation, 100 testing), the SysCaps-nl model uses prompt augmentation to increase both the quantity and quality (e.g., higher variability in syntax and style) of the training data. On the buildings dataset, SysCaps-nl does not use prompt augmentation. This is because the size of the buildings dataset was too large to create multiple captions per building. We expect that training the SysCaps-nl model on the wind dataset without prompt augmentation would result in similar (or worse) performance than SysCaps-kv. We will run this experiment and add the results.

• [W5 - non-experts]: We agree that our method has the potential to help explain system attributes for surrogate models to non-experts. The current manuscript lacks discussion about this, which we will rectify. We emphasize, however, that evaluating this goes beyond the scope of our current methodology. For example, we imagine this would require instructing the LLM to identify system attributes whose names might be inscrutable to humans, and to then use the rest of the metadata and system prompt to come up with better names that are easier for non-experts to use. Then, the LLM could be instructed to use these new names to create the SysCaps. We will highlight this when discussing future work and potential impacts in our Conclusion.

• [Q1 - Template sentences]: We did not try training with template sentences. However, we consider the “key-value” captions used by the SysCaps-kv models, which are lists of attribute names and value key1:value1 | key2:value2 | … as an estimate for the performance of sentence templates. We believe this is a reasonable assumption because a sentence template just adds extra words which do not possess new information about the system being simulated.

• [Q2 - RFE]: There are only 5 attributes for the wind dataset, which is relatively few and so RFE is not needed. The captions generated by the LLM for the wind dataset are also relatively short.

• [Q3 - augmenting training] Based on the results of our prompt augmentation experiment (Section 6.5), we have evidence that, yes, increasing both the quantity and quality of the training data via augmentation would lead to better performance. This can include generating extra captions by prompting the LLM to create captions using attribute synonyms. We will add more qualitative examples of the LLM-generated captions to the appendix in the revised PDF. Here is an example where the LLM paraphrased (bolded):

  • "The wind plant is designed with a cluster layout, featuring 40 turbines of varying heights. Each turbine boasts a rotor diameter of 130 meters, providing ample sweep area to capture the wind's energy. With a mean turbine spacing of 7 times the rotor diameter, the plant is optimized for efficient energy production, while minimizing visual impact and land usage.”
  • “The wind plant features a layout of multiple strings, with 73 turbines standing tall at an impressive rotor diameter of 130 meters. The turbines are spaced at an average distance of four times the rotor diameter, resulting in a highly efficient and productive wind farm. Each turbine is equipped with a rated power of 3.4 megawatts, making it capable of generating a significant amount of electricity from the wind.”

For Question 4, perhaps you can clarify—is the question about whether we could improve performance by using a threshold to filter out “low quality” captions?

Dear Reviewer 38tw,

Thank you for your review! A single fine-tuned language-model (LM) for timeseries regression would be appealing. However, we can justify our choice to instead use two separate components for encoding text and performing timeseries regression.

• [Limited context window size of LMs]: A single simulation for a complex system may run for thousands of time steps. For example, the building surrogate models in our experiments predict hourly energy consumption for 1 year (T= 8,760). This hinders the use of language models—BERT has a context window size of 512, and even alternative models such as LongFormer and Llama-2-7b have a max context window size of 4,096. Of course, there exist advanced tricks to aggregate multiple embeddings computed for a single document where the document length exceeds the context window. We believe the RNNs and State Space Models (SSMs) we use offer a simpler solution as they have no such limitations on the sequence length.
• [Computational expense of LMs]: The quadratic complexity of the self-attention mechanism in Transformer-based language models, as well as the high number of parameters in LLMs such as Llama-2-7b, makes LMs poor choices for surrogate models of complex systems in practice. For a concrete example, to conduct the sensitivity analysis case study (Figure 4), we perform 960K model predictions, which took us ~1 hour on 1 NVIDIA A100-40GB GPU. If we generously assume that Llama-2-7b takes 10 seconds to generate 8,760 tokens (this is assuming a generation speed of 876 tokens/sec), this case study would have taken 9.6M seconds, or ~111 days, to run. Our multi-component approach helps make our framework ready to be used by practitioners.
• [Our two-step process vs. one-step]: In our framework, at train time the first step is to use an LLM to generate synthetic natural language SysCaps. The second step is to encode the text caption for multimodal timeseries regression. The first step is only used at training time---at test time, a person can interact with the multimodal surrogate model and prompt it with a text description. While the idea of using a single LLM to directly perform timeseries regression is intuitive, as previously stated, we intentionally want to avoid using an LLM as a text encoder due to the computational burden. Moreover, an additional advantage of our approach is in its modularity. We can decide to swap out the Llama-2-7b LLM for a more powerful LLM such as Claude or GPT-4 to generate training captions. This is not so easy if the LLM is also fine-tuned for timeseries regression. Likewise, we can swap out the lightweight text encoder model for a more powerful one. Our experiments comparing DistillBERT and BERT (Table 1) show that larger, more powerful text encoders lead to better regression accuracy.

For the question about a comparison between time-series centric models and generic architectures, could you give us a specific model that you have in mind? The exact comparison you are requesting is unclear to us.