OWL: A Large Language Model for IT Operations

Q1: limited novelty

A1: The prevailing understanding within the AI community is that general-purpose Large Language Models (LLMs) such as GPT-4 exhibit limitations when tasked with domain-specific challenges. Attempts to employ prompt engineering as a means to infuse expert knowledge into LLMs and steer them toward providing solutions have yielded inconsistent and suboptimal outcomes. This has led to a collective agreement on the necessity of constructing large-scale models that are intricately tailored to specific vertical domains, ensuring a higher degree of precision and effectiveness in their performance[1][2]. The development of our model first represents a dedicated endeavor tailored to the nuances of IT operations and maintenance (O&M) in specific domains. It is the culmination of meticulous expertise and specialized knowledge. The data that fuels this innovation is of immense value—our carefully curated fine-tuned dataset, along with the benchmark data, is a testament to the labor-intensive process of manual labeling and collection, achieved without the crutch of data augmentation techniques. Besides, our approach is generalizable and can be easily migrated to other fields. The originality and ingenuity of our approach have not gone unnoticed, garnering acclaim from our peers in the field. Our innovativeness is also recognized by other reviewers.

Reference:

[1] https://github.com/luban-agi/Awesome-Domain-LLM

[2] BloombergGPT: A Large Language Model for Finance

W3: Duplications and more convincing evaluation

A3: Our benchmark and instruction dataset have no overlap. To ensure data quality, we have employed the MinHash method[5][6] for deduplication. MinHash is used for estimating the similarity between datasets and is particularly useful in large-scale clustering problems. The main idea is to reduce the dimensionality of the datasets by converting large sets into smaller fingerprints while preserving the similarity information.

Besides, For more convincing results, we also provide a human evaluation on the Q&A test. For the Q&A test, we supplement it with human scoring and provide the Pearson coefficient between the human evaluation scores and the GPT-4 scores. For the single-mode evaluation, we adopt a cross-validation approach to ensure that each answer is scored by at least three people who are all experienced operation and maintenance engineers, and in the end, we take the average score as the human score. The Pearson coefficient is 0.78 and P-Value is 0.065. Generally, when the coefficient is greater than 0.7, it is usually considered to be a strong correlation. The detailed average scores are below:

For pairwise-mode scoring, we use the results from GPT-4 evaluation as a reference for the evaluators. We also adopt a cross-validation approach to ensure that at least three people check each comparison result. If the GPT-4 score is incorrect, annotators must provide a new comparison result and give reasons; if all annotators agree that there is no issue with the scoring result of a particular question, then that result is kept. For inconsistent scoring results, we engage in further discussion and apply a majority vote system. The human scoring results are shown below and the Pearson correlation coefficient for the win rate is 0.97.

GPT4 result：

Result after manual review：

Reference:

[5] https://mattilyra.github.io/2017/05/23/document-deduplication-with-lsh.html

[6] https://github.com/ekzhu/datasketch

Thank you once again for your insightful feedback. We will incorporate these experimental updates in the revised version of the paper.

W2: More baselines on the long-context inference experiment.

A2: Although NBCE has not been published, it is also a recognized method proposed by the author of RoPE. Such methods do not require training or structural changes to the model, nor do they require interpolation of the positional embedding to extend the context. We have selected more methods for comparison. The test questions are the same extracted from IT domains. The perplexity (PPL) results:

Reference:

[3]https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

[4]Parallel Context Windows for Large Language Models

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.

W3. Content distribution

A3: We will incorporate the related work section into the main text and include more background information on IT tasks in the new version.

W2: Human evaluation on Q&A test

A2: Please refer to Q1

Q2&Q3: Effectiveness and Robustness of OWL on downstream tasks.

A2&A3: Compared to traditional algorithms, OWL can handle more complex scenario questions and provide interpretable reasons, which plays an important role in IT-related tasks such as root cause localization. This can help operation and maintenance personnel better analyze and observe the system. In addition, traditional models require a large amount of labeled data and retraining to solve specific scenario problems, while OWL does not require more training and performs better in zero-shot scenarios. We demonstrate the effectiveness and robustness of OWL with results on more downstream tasks.

We select a new downstream task for a more complex scenario test, Trace-Log Anomaly Detection, which is similar to a log anomaly detection task. This task uses two types of data, logs, and traces, for system anomaly detection and fault localization. We align with the experimental setup of DeepTraLog [1] and adopt the same dataset, TrainTicket[2]. In the experiments, we input the two types of mixed data directly into OWL, allowing the model to determine on its own whether there is an anomaly. The results comparison show the effect of OWL on such fault localization analysis:

Reference:

[1] DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning

[2] Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study

[3] Unsupervised Detection of Microservice Trace Anomalies through Service-Level Deep Bayesian Networks

[4] Anomaly Detection from System Tracing Data Using Multimodal Deep Learning

Thank you once again for your insightful feedback. We will incorporate these experimental updates in the revised version of the paper.

W1: Limited novelty

A1: The prevailing understanding within the AI community is that general-purpose Large Language Models (LLMs) such as GPT-4 exhibit limitations when tasked with domain-specific challenges. Attempts to employ prompt engineering as a means to infuse expert knowledge into LLMs and steer them toward providing solutions have yielded inconsistent and suboptimal outcomes. This has led to a collective agreement on the necessity of constructing large-scale models that are intricately tailored to specific vertical domains, ensuring a higher degree of precision and effectiveness in their performance[1][2]. The development of our model first represents a dedicated endeavor tailored to the nuances of IT operations and maintenance (O&M) in specific domains. It is the culmination of meticulous expertise and specialized knowledge. The data that fuels this innovation is of immense value—our carefully curated fine-tuned dataset, along with the benchmark data, is a testament to the labor-intensive process of manual labeling and collection, achieved without the crutch of data augmentation techniques. Besides, our approach are generalizable and can be easily migrated to other fields. The originality and ingenuity of our approach have not gone unnoticed, garnering acclaim from our peers in the field. Our innovativeness are also recognized by other reviewers.

Reference:

[1] https://github.com/luban-agi/Awesome-Domain-LLM

[2] BloombergGPT: A Large Language Model for Finance

Q1: Human evaluation on Q&A test

A1: For the Q&A test, we supplement it with human scoring and provide the Pearson coefficient between the human evaluation scores and the GPT-4 scores. For the single-mode evaluation, we adopt a cross-validation approach to ensure that each answer is scored by at least three people who are all experienced operation and maintenance engineers, and in the end, we take the average score as the human score. The Pearson coefficient is 0.78 and P-Value is 0.065. Generally, when the coefficient is greater than 0.7, it is usually considered to be a strong correlation. The detailed average scores are below:

For pairwise-mode scoring, we use the results from GPT-4 evaluation as a reference for the evaluators. We also adopt a cross-validation approach to ensure that at least three people check each comparison result. If the GPT-4 score is incorrect, annotators must provide a new comparison result and give reasons; if all annotators agree that there is no issue with the scoring result of a particular question, then that result is kept. For inconsistent scoring results, we engage in further discussion and apply a majority vote system. The human scoring results are shown below and the Pearson correlation coefficient for the win rate is 0.97.

GPT4 results：

Results after manual review：

Thanks the authors for their response. I have read it but I think it does not convince me to change the rating.

W3. Typos

A3: Thank you for your insightful feedback. We will improve it in the revised version of the paper.

W2: The relationship between long-context inference and OWL.

A2: The challenge of long-text input is something every large language model encounters, and in the IT operations domain, there are instances where long text or document input is required. Such situations may arise during the review of system logs, the analysis of extensive error reports, or the processing of detailed technical documentation. To handle these scenarios effectively, techniques such as NBCE/HMCE enable the model to maintain context over longer stretches of text and provide more accurate and relevant responses, thus enhancing the model's utility in complex IT operational tasks.

Q2: Unfair comparison in Figure 3

A2: In Figure 3, OWL is obtained by conducting MoA training on LLaMA2. Other models do not undergo MoA training. Consistent with the comparison method in Figure 3, we present the experimental results of different baseline models using MoA for comparison. The experiments show: 1. The effectiveness of MoA. 2. The stronger the base capabilities of the model, the better the results after using MoA.

Thank you once again for your insightful feedback. We will incorporate these experimental updates in the revised version of the paper.

Q1: Tokenization

A1: We retrain the embedding layer of the LLaMA2 model, while other layers are frozen. After increasing the vocabulary size, the dimensions of the token embeddings changed. We first randomly initialized the embedding layer, and then retrained it during the instruction-tuning stage, whereas the parameters of the other layers of the base model are frozen and remain unchanged.

W1: More results on HMCE and MoA methods.

Ablation 1: More baselines on the long-context inference experiment.

The test questions are the same extracted from IT domains. The perplexity (PPL) results:

Ablation 2：Generalizability of HMCE.

In the field of IT operations, there are many issues that involve extensive text, hence the need to extend long text input. To verify the generalizability of HMCE, we conduct experiments on PG19 dataset [1] to test perplexity under different input lengths. The experiments have well demonstrated the effectiveness and generalizability of HMCE.

Ablation 3：Different Rank in MoA training.

We conducted ablation experiments on the rank parameter in LoRA within MoA training. Results on the multiple-choice questions in owl-bench:

Reference:

[1] Compressive transformers for long-range sequence modelling

[2]https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

[3] Parallel Context Windows for Large Language Models

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.

Q1: Human evaluation on Q&A test

A1: For the Q&A test, we supplement it with human scoring and provide the Pearson coefficient between the human evaluation scores and the GPT-4 scores. For the single-mode evaluation, we adopt a cross-validation approach to ensure that each answer is scored by at least three people who are all experienced operation and maintenance engineers, and in the end, we take the average score as the human score. The Pearson coefficient is 0.78 and P-Value is 0.065. Generally, when the coefficient is greater than 0.7, it is usually considered to be a strong correlation. The detailed average scores are below:

For pairwise-mode scoring, we use the results from GPT-4 evaluation as a reference for the evaluators. We also adopt a cross-validation approach to ensure that at least three people check each comparison result. If the GPT-4 score is incorrect, annotators must provide a new comparison result and give reasons; if all annotators agree that there is no issue with the scoring result of a particular question, then that result is kept. For inconsistent scoring results, we engage in further discussion and apply a majority vote system. The human scoring results are shown below and the Pearson correlation coefficient for the win rate is 0.97.

GPT4 results：

Results after manual review：

Thank you once again for your insightful feedback. We will incorporate these experimental updates in the revised version of the paper.

Q1: Limited novelty

A1: The prevailing understanding within the AI community is that general-purpose Large Language Models (LLMs) such as GPT-4 exhibit limitations when tasked with domain-specific challenges. Attempts to employ prompt engineering as a means to infuse expert knowledge into LLMs and steer them toward providing solutions have yielded inconsistent and suboptimal outcomes. This has led to a collective agreement on the necessity of constructing large-scale models that are intricately tailored to specific vertical domains, ensuring a higher degree of precision and effectiveness in their performance[1][2]. The development of our model first represents a dedicated endeavor tailored to the nuances of IT operations and maintenance (O&M) in specific domains. It is the culmination of meticulous expertise and specialized knowledge. The data that fuels this innovation is of immense value—our carefully curated fine-tuned dataset, along with the benchmark data, is a testament to the labor-intensive process of manual labeling and collection, achieved without the crutch of data augmentation techniques. Besides, our approach is generalizable and can be easily migrated to other fields. The originality and ingenuity of our approach have not gone unnoticed, garnering acclaim from our peers in the field.

Reference:

[1] https://github.com/luban-agi/Awesome-Domain-LLM

[2] BloombergGPT: A Large Language Model for Finance

Thank you for your detailed responses to my questions.

I'm overall positive about this paper and I'd like to keep my scores. Please try to incorporate your responses here into your future versions. Thanks!