From Commands to Prompts: LLM-based Semantic File System for AIOS

W4: One more thing the paper lacked was elaborating on the use cases where this architecture can be used.

Q5: What are the other use cases of this architecture in real life scenarios?

Semantic-based file systems are designed to cater to a broad range of usecases. For instance, as highlighted in [1], traditional file systems often impose a cumbersome approach to organizing documents, posing significant challenges for small and medium-sized enterprises (SMEs), public administration bodies, and individual users. LSFS addresses this gap by enabling more efficient organization and management of document content, reducing operational complexity for non-computer practitioners, including SMEs and public administration agencies. Furthermore, [2] identifies semantic file systems as a critical development trend, emphasizing their versatility and applicability across various domains in information technology (IT). Additionally, LSFS can fill a significant gap in current LLM-based multi-agent systems, as noted in [3], which often lack robust mechanisms for managing interaction records and background knowledge. By providing a more intuitive file management framework, LSFS benefits both end-users and system designers by enhancing file access and reducing administrative overhead.

Q3: Are there any security concerns for using this architecture?

Thank you for raising the question. We designed several mechanisms to address potential security concerns:

1. We added a process lock to LSFS to prevent consistent reads and writes to the same file
2. We design a user confirmation step: When a user makes a change to a file, the user will be asked to confirm the changed object twice
3. We designed rollback operations: if the user makes a wrong change to the file, they can roll back to the correct version

For first and third mechanisms, the file operation reliability can be improved as long as these two mechanisms are enabled.

For the second mechanism, we conducted a quantitative evaluation of two aspects: the probability of file misoperation and the proportion of risky operations. We compared these metrics with and without the confirmation mechanism enabled.

In the current experiment, we used the retrieval function to locate target files for operations. The table below shows the probability of retrieval errors with and without the confirmation step:

Additionally, we evaluated the proportion of potentially dangerous operations executed (e.g., write, update, or delete) across all file management APIs. The results below demonstrate that the confirmation mechanism in LSFS effectively prevents unintended dangerous operations:

These results highlight that enabling the confirmation mechanism significantly enhances the safety and reliability of file operations in LSFS.

Q6: This seems like an ongoing problem that needs to be resolved; are there any similar existing architectures? Have you looked at those papers?

As highlighted in our related work, existing semantic file systems integrate semantics only into metadata and do not leverage large models to create a fully semantic file system.

To the best of our knowledge, no work has systematically proposed semantic file management based on LLMs. Existing LLM-agent systems focus on enhancing agent functionalities through leveraging file content while neglecting the fundamental file management based on semantics. Agent frameworks with storage mechanism enabled, such as Autogen[4], Camel[5], and OS-Copilot[3], build user profiles and obtain knowledge from files. Knowledge in these systems typically stores agents' past interactions or acquired different files. However, agent systems built by these frameworks face limitations due to their reliance on traditional file management methods. One significant challenge is the need for developers to explicitly set up file paths for agents, which becomes increasingly cumbersome as the number of agents and agent-related files (e.g., task completion records and knowledge bases) grows. This limitation hinders scalability and efficiency in deploying multiple agents. Recognition of the limitation in existing agent systems inspires us to propose a semantic file system that supports the development of LLM-based agents, enabling more efficient and scalable semantic file management to support building of agents.

Q7: Is there an Andon Cord mechanism to stop the LLM to give out hallucinations and wonky results to the user?

First, our parser helps mitigate the hallucination problem by incorporating the fault-tolerance mechanism. If the LSFS parser incorrectly parses a user's command and fails to map it to the API, the failure case will be added into the prompt like the result of {failure case} is wrong, you should refer to the case and regenerate it. Then let LSFS parser generate the answer again. Secondly, our rollback mechanism provides a safeguard against potential LLM hallucinations during file operations. If the LLM erroneously modifies a file, the user can restore it to its previous version using the rollback feature.

Q8: While scanning through the files, is the data saved in memory? Does the data contain PII information (ppersonal information about the user)?

In the LSFS architecture, large language models (LLMs) serve as downstream processing tools. Users cannot add files to LSFS through LLMs, nor does LSFS provide interactive memory functionalities for LLMs. Consequently, files scanned or referenced during LLM interactions are not automatically added to the LSFS system's memory. Instead, the files present in LSFS are explicitly added by the user. Besides, to help protect the personal information of user, we provide the user confirmation mechanisms, which is detailed in our response to Q1.

[1] D. Di Sarli, F. Geraci, “GFS: a Graph-based File System Enhanced with Semantic Features”, Proceedings of the 2017 International Conference on Information System and Data Mining, pp. 51-55, April 1-3, 2017

[2] Mashwani, S.R. and Khusro, S. 2018. The Design and Development of a Semantic File System Ontology. Engineering, Technology & Applied Science Research. 8, 2 (Apr. 2018), 2827–2833. DOI:https://doi.org/10.48084/etasr.1898.

[3] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024

[4] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel:Communicative agents for "mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2023.

[5] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li,Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.

Thank you very much for taking your valuable time to review our paper and provide us with many constructive suggestions. Below are our detailed responses to your concerns:

W1: The paper could have used an example to walk through the implementation. Each component description could have been presented with design diagrams or a flowchart that is easy to understand; visual representation always helps!

Thank you for raising this question. A finer-grained example to walk through the components has been added in the Fig.1. in the revised version.

W2: More evaluations to prove their architecture is better than the traditional ones based on performance, latency, operational burden, and cost.

Q2: How much cost are we saving with the new architecture?

Q4: How much of an operational overhead is this architecture based on traditional architecture?

Compared to traditional file systems, LSFS offers enhanced functionality, such as grouping and semantic retrieval, and performs better in keyword-based retrieval tasks. However, LSFS introduces additional overhead, including LLM inference and database operations, leading to longer execution times compared to traditional systems. Despite this, LSFS simplifies operations by reducing the time users spend learning and inputting commands. To provide a clearer understanding of the time costs, we present a comprehensive time analysis below.

In order to quantitatively analyze the time difference between traditional file systems and LSFS, we invited 10 Ph.D. to execute the above commands with Linux commands in traditional file systems and natural language commands in LSFS, and calculated the time consumption of each part according to our above time division. The results are as follows:

LSFS

Traditional FS

W3: The paper didn't touch on any security concerns while using the LLMS. Are there guardrails in place to restrict the LLMs to scanning through the personal data?

Q1: Are there guardrails in place to restrict the LLM to not scan through personal data?

In LSFS, our user filter is implemented to prevent LLMs from scanning to personal information. When LSFS retrieves an eligible file to perform subsequent tasks, it will first send the user confirmation, and if the user finds that the retrieval results contain personal information, he can cancel sending the file to LLM.

Q9: In §5.2 still, what about very large number of files?

We increased the number of files to 120, including binary file types ".pdf ", ".doc" and plain text files ".txt". The experimental results are as follows:

As shown in the table and Tab. 2, retrieval time increases with the number of files, but LSFS maintains a linear growth trend and offers more stable retrieval accuracy compared to the pure LLM method. This demonstrates the scalability of LSFS. However, at very large scales, system overhead may further delay retrieval time and we plan to evaluate our system's performance under such large-scale scenarios in our future work.

Q10: Ibid., same for the number of versions?

In our experiments, we tested up to 40 versions and observed that the rollback time does not increase exponentially with the number of versions rolled back, as shown in Fig. 6. This is because the rollback API stores each file version independently, allowing efficient retrieval. However, when the number of files scales up significantly, the rollback time could still increase due to heavier file storage overheads will affect the whole system's latency and exploration on large-scale scenarios in our future work.

[1] D. Di Sarli, F. Geraci, “GFS: a Graph-based File System Enhanced with Semantic Features”, Proceedings of the 2017 International Conference on Information System and Data Mining, pp. 51-55, April 1-3, 2017

[2] Mashwani, S.R. and Khusro, S. 2018. The Design and Development of a Semantic File System Ontology. Engineering, Technology & Applied Science Research. 8, 2 (Apr. 2018), 2827–2833. DOI:https://doi.org/10.48084/etasr.1898.

[3] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024

Q3: Could you add a positioning sentence in §2.3 to explain clearly where LSFS delineates from this research axis?

We appreciate this feedback. In response, a positioning statement has been added: "While researches of agent systems primarily focus on build of LLM applications that can leverage file resources, our represents a fundamental innovation in the infrastructure that manages file resources based on semantics to support LLM-based agent systems."

Q4: In this stage many operations from traditional FS aren't there, from what I understood, this is typically the case for right modification of files or group affiliation… These would be particularly helpful so to "propagate" these rights to any retrievers, preventing typically to see the private/exclusive file of someone else appearing in my search results, wouldn't it?

Private information is important for file system. Our LSFS system operates as a middleware layer over a traditional file system, inheriting all file permissions from the underlying system, ensuring no file has multiple user affiliations. To further safeguard user privacy, LSFS incorporates a defensive mechanism that prevents large language models (LLMs) from accessing personal data without explicit consent. When a user requests file access through LSFS, the system prompts for confirmation before sharing data with the LLM. Users retain control, with the option to cancel the operation at any point, thereby preventing unintended data exposure.

Q5: In §4,2 (line 294), the authors mentioned that supervisor updates "periodically" what is the period between each check and therefore how expensive resource-wise is it? Did the authors check various values for this, searching for the sweet-spot between resource-consumption and freshness of the LFSF data? Also how does it scale in terms of file number and disk footprint?

The LSFS supervisor checks for file updates every 10 seconds. To assess its resource usage and scalability, we measured its response time across different file counts and monitored its CPU usage. The results are as follows:

CPU Usage: Consistently between 0.1% and 0.2%.

These results show that the supervisor is efficient, with millisecond-level response times and low CPU usage remained, even as the number of files increases.

Q6: Overall, §4.4 seems to be more or less an NL2CSV tool, filling fields of a JSON, right? In such case, this is something that the community has been exploring a lot these past two years, so maybe adding some pointers wouldn't hurt. This goes also for the §5.1 associated with RQ1.

Thank you for mentioning the related works. References of related works have been updated in the revised version.

Q7: Are authors considering releasing their test data for §5.1? Also, it would be good to have some examples in the body of the article.

The test data has been included in our anonymous code link. An example of the natural prompt command is presented in Fig.1. and more examples of the test data are given in the appendix due to the space limit.

Q8: In §5.2 why no QWen or Gemma in the experimental run for Table 2, and no Gemma in Figure 6?

We conducted the above experiments for both Qwen-2 and Gemma-2 but excluded these models from the final results due to their poor performance on the task and hallucinated outputs. For example, in Table 2, when judging files without the LSFS system, both Qwen-2 and Gemma-2 produced outputs containing inaccuracies. Using identical text and prompts to check for keyword1 and keyword2, their responses included irrelevant stories about Michael Jordan. Similarly, in Figure 6, Gemma-2 demonstrated unstable outputs. This instability makes it challenging to assess the model's reliability and to derive meaningful conclusions from the system's performance, and the discussion of this has been updated in the revised version.

4. Very disappointed to have the future directions (as Appendix H) not included at all in the main article, but instead, pushed as optional reading material at the very very end of the .pdf…

Thank you for recognizing the future directions that can be built on our method. The part of future directions has been added in our revised version in Section 6.

5. Finally, the overall article, from §3 onwards especially, reads a bit like a technical report and lacks -to me- a step-back to better highlight the novelties and the perspectives while guiding the reading with more examples / intuitions directly in the text.

We would like to highlight our work addresses a fundamental problem in agent systems: the lack of a systematic approach to semantic file management and would like to present the two major novelties we propose when designing this system as the following.

• The first novelty is we close the semantic gap from prompts contain file indentions to actual file operations. Traditional file systems rely on exact matches and rigid hierarchical structures, making it difficult to perform operations based on semantic understanding such as 'find documents about machine learning'. To address this challenge, we propose the semantic-based file index structure using vector databases to capture and store file content meanings. Also, we design the semantic parser that accurately translates semantic queries into new designed file operational APIs.
• The second novelty is that simple architecture designs that directly map from language prompts to file system calls can pose safety vulnerabilities. For instance, a misinterpreted command could lead to unintended file deletions or destructive overwrites, potentially causing irreversible damage. To address this challenge, we design the layered architecture with high-level file operation API to isolate the direct access to low-level file syscalls. We also design multiple verification mechanisms at both APIs and syscalls to validate operations before execution and design the rollback mechanism that can reverse potentially harmful operations, ensuring operation safety.

Response to Minor Comments:

Thank you for your valuable writing suggestions. The content has been updated accordingly in our revised version.

Response to Questions

Q1: Intro (line 57) "The community still lacks a more general LSFS to serve as a common foundation that can be used by various agents on the application-level." Do you have some references backing up this lack? I mean, have people expressed somewhere they'd like/need a FS organized by an intelligent agent, i.e., an LLM here?

Thank you for your question of the use cases for our LSFS. Semantic-based file systems are designed to cater to a broad range of users. For instance, as highlighted in [1], traditional file systems often impose a cumbersome approach to organizing documents, posing significant challenges for small and medium-sized enterprises (SMEs), public administration bodies, and individual users. LSFS addresses this gap by enabling more efficient organization and management of document content, reducing operational complexity for non-computer practitioners, including SMEs and public administration agencies. Furthermore, [2] identifies semantic file systems as a critical development trend, emphasizing their versatility and applicability across various domains in information technology (IT). Additionally, LSFS can fill a significant gap in current LLM-based multi-agent systems, as noted in [3], which often lack robust mechanisms for managing interaction records and background knowledge. By providing a more intuitive file management framework, LSFS benefits both end-users and system designers by enhancing file access and reducing administrative overhead.

Q2: Related Work (line 133) "Besides, it integrates comprehensive semantic information across all aspects of the file system–from storage and file operations to practical applications" This sentence is a bit vague, could you name some of the semantic information here, so to guide the reader?

In LSFS, "comprehensive semantic information" reflects our focus on integrating semantic insights across various stages of file management. Examples include:

1. Semantics for File Storage: Beyond traditional metadata (size, timestamp), LSFS adds details like themes and keywords to enrich file descriptions.
2. Semantics for File Content: Each file is indexed semantically, enabling operations like finding a file based on descriptors such as "a Hollywood science fiction movie." Files can also be grouped by related topics.
3. Semantics for File Operations: LSFS includes a parser that translates natural language commands into specific actions using semantic understanding.

Thank you very much for providing us with many constructive suggestions. Below are our detailed responses to your concerns:

Response to General Remarks

1. Even though the Introduction is clear, I'd have liked a more concrete / detailed example, maybe having a finer-grained figure 1 would help.

Thank you for this suggestion. A finer-grained figure 1 has been updated in our revised version of paper.

2. The balance between §3 Vs. §4 is unexpected, I would have imagined a more detailed architecture section (§3), explaining the various design choices. Instead, the authors motivated their architecture. In addition to not being referenced, this motivation -to me- would have been better positioned directly in the Introduction. Similarly, the overview of the architecture and the description of Figure 2 would have benefit also the introduction of an example, especially since Fig2.a doesn't contain precise information, but rather e.g. a list of blocks entitled API.

Thank you for your suggestion of the structure of the paper, we have updated all of them to the paper and a detailed example and figure has been added to §3.

3. Overall, for §5, it would have been very interesting and convincing to see an experiment involving users' usage and performances, using a TFS and the presented LSFS. The authors could have reviewed the success rate and the time efficiency of users in both settings together with collecting feedback from them, following the traditional user studies.

Thank you for the suggestion of collecting user feedback. The comparison of keyword retrieval accuracy of users and LSFS can be seen in Table.3. In addition, we invited 10 Ph.D students to supplement the following experiments to evaluate the time cost to fully complete the task:

Traditional command execution typically follows this workflow: learn command parameters (via ChatGPT or search engines) > locate appropriate file paths through OS lookup mechanisms > input the complete command > and finally obtain results. In contrast, LSFS streamlines this process into a simpler workflow: input natural language commands with file names or semantic keywords, which are then processed by the LSFS parser for task execution.

In order to quantitatively analyze the time difference between traditional file systems and LSFS, we adopt FS commands and LSFS commands to achieve different file operations shown below. 10 Ph.D. student are invited perform file operations through using Linux commands in traditional file systems and using natural language commands in LSFS, respectively.

The breakdown of time in different steps for the students to operate files correctly is collected and the average time in each step is calculated as below.

LSFS

Traditional FS

Although LSFS takes longer for executing file operations due to the inference time of the LLM, the total time it takes for users to successfully perform the file operations is significantly shorter than that of traditional file systems. This is because traditional systems require users much more time to learn through trial-and-error during the "Learn Command" step. In contrast, LSFS simplifies the process through natural language commands, reducing overall time.

Thanks for your suggestions:
As you pointed out, the time required for the learn command tends to approach zero over the long run. However, the time a user spends locating the relevant path and entering commands remains long. Additionally, as demonstrated in Table 4 of the paper, even graphical user interfaces such as Spotlight often return numerous irrelevant files, requiring further filtering by the user, which incurs additional time costs. Our results show that, even after subtracting the learning time, traditional file systems still take significantly more time to complete tasks compared to LSFS. However, your suggestions are very valuable to us and we plan to conduct a larger-scale user study covering users with different expertises in computer use in the future exploration.

We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns:

W1: weak evaluation, based on examples and not on more extensive use cases and human evaluation, evaluation setup not described in detail / cannot be reproduced

We would like to further clarify our experiment setting as below.

1. In Experiment 5.1, we validate the effectiveness of our parser using four different LLM backbone. For each API, we created 30 use cases, each with a unique language structure. The test data is provided in the anonymous link.
2. In Experiment 5.2, we evaluated the effectiveness of LSFS in semantic retrieval. Since traditional file systems lack semantic retrieval capabilities, we used LLM alone as the baseline for file retrieval in our evaluation. Specifically, we measured the time and accuracy of executing a semantic file retrieval task by simply feeding file content into LLM to retrieve files and retrieving files using LSFS. Our target files contain various types of files such as ".txt,.pdf,.doc", etc. To make our experiments more extensive use cases, we supplemented with experiments under 120 files, and the results are as follows:

3. In Experiment 5.2, in order to check the robustness of our rollback function when the number of files increases, we tested the rollback time under 5-40 versions respectively, and obtained the robustness of the rollback time.

4. In Experiment 5.3, we compared LSFS with traditional file retrieval methods based on Precision, Recall, and F1 score. The methods include:

   • TFS-search-window: Uses the computer's search window (e.g., MacOS Spotlight) to retrieve both binary and plain text files.
   • TFS-grep: Uses the Linux terminal command grep, which can only retrieve plain text files.
   • TFS-grep*: An enhanced version of TFS-grep that first converts binary files into plain text before using grep to retrieve.

5. In the second part of Experiment 5.3, we compared LSFS and pure prompting to different LLMs as baselines to generate code for creating sharable file links.

Finally, we would like to highlight that our code is provided through the anonymous link and it's all reproducible.

W2: unclear for which users this could be helpful

Semantic-based file systems are designed to cater to a broad range of users. For instance, as highlighted in [1], traditional file systems often impose a cumbersome approach to organizing documents, posing significant challenges for small and medium-sized enterprises (SMEs), public administration bodies, and individual users. LSFS addresses this gap by enabling more efficient organization and management of document content, reducing operational complexity for non-computer practitioners, including SMEs and public administration agencies. Furthermore, [2] identifies semantic file systems as a critical development trend, emphasizing their versatility and applicability across various domains in information technology (IT). Additionally, LSFS can fill a significant gap in current LLM-based multi-agent systems, as noted in [3], which often lack robust mechanisms for managing interaction records and background knowledge. By providing a more intuitive file management framework, LSFS benefits both end-users and system designers by enhancing file access and reducing administrative overhead.

W3: unclear how robust the system is when translating NLP input into actual actions.

Q2: Is the accuracy of the LSFS parser of about 90% enough for meaningful work? That means 10% incorrect results. How did you collect the NLP commands for this evaluation?

We conduct the case study of the incorrect results and find that LSFS parser sometimes performs bad in parsing complex commands due to the capability and inherited randomness of the LLM. To further improve the reliability of our system, we make the following enhancements. We add the failure case to the prompt like the result of {failure case} is wrong, you should refer to the case and regenerate it. Then let LSFS parser generate the answer again, the experiment results as follow:

Gemini-1.5

GPT-4o-mini

Qwen2:7b

Gemma:2b

After a second parsing using the use cases that were incorrectly parsed the first time, we can see that all LLM backbone achieved 100% accuracy on each task except Gemma-2, which did not achieve 100% accuracy on the Retrieve-Summary API. This means that our parser can parse the task correctly at most twice on most tasks and llm backbone, so we consider the parser to be useful for mapping work. For data collection, we invited 10 Ph.D. students to write preliminary instructions for related tasks. Then the results are used for testing, and the instruction information is fine-tuned according to the test results, and finally the instruction with the highest score is selected.

W4: unclear how the new API maps and extends conventional file handling APIs, and why setting up a new API set is superior to adding some APIs to a conventional file system.

Q1:Your systems seems to have advantages for searching and retrieving files based on keyword or semantic search. This could be implemented on top of a conventional file system, why implement a new API for that?

Thank you for the recognition of the advantages of our LSFS. The building of APIs by simply combining system calls to achieve functionalities can not consider the semantic information of file in the file's lifecycle including creation, update, deletion. Therefore, we can not use them to achieve flexible semantically related APIs. Under this consideration, we design a new set of APIs to build the semantic index of file and support more flexible semantic file management operations.

Q3: How exactly did you perform the evaluation summarized in table 2?

In the "w/o LSFS" scenario, we input each text directly into the LLM sequentially, checking if it meets the target condition. If the condition is satisfied, the subsequent action is executed; otherwise, the text is skipped. In contrast, under the "w/ LSFS" setup, LSFS performs an initial search and prompts the user for confirmation. The corresponding file from the search results is then directly fed into the LLM for further processing. Additional details can be found at Appendix D.

[1] D. Di Sarli, F. Geraci, “GFS: a Graph-based File System Enhanced with Semantic Features”, Proceedings of the 2017 International Conference on Information System and Data Mining, pp. 51-55, April 1-3, 2017

[2] Mashwani, S.R. and Khusro, S. 2018. The Design and Development of a Semantic File System Ontology. Engineering, Technology & Applied Science Research. 8, 2 (Apr. 2018), 2827–2833. DOI:https://doi.org/10.48084/etasr.1898.

[3] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024

Dear reviewer a3yw,

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. As the deadline for the extended discussion period is nearing, in order for us to have sufficient time to address any additional questions you may have, we kindly encourage you to engage in the ongoing discussion and share any further insights or clarifications you may have.

Thank you very much for your time. We look forward to hearing from you soon.

Best Regards,

All authors

W4. The experimental results need more explanation. The traditional file system needs to execute commands to extract files, which should be faster than calling one LLM, even though these LLMs are light-weight. The authors should also list the inference time of LLMs, which should be also counted as the interaction time between users and the file system. Then the authors can also list the time that users manually write commands, which should be a good point to prove the point that -- LSFS can not only speed up file retrieving accuracy and speed, but also can reduce the interaction time between users and file system.

While LSFS operates as an intermediate layer for traditional file systems, its file operations exhibit longer latency compared to conventional systems due to LLM inference and vector database retrieval times. Despite this, LSFS simplifies operations by reducing the time users spend learning and inputting commands. To provide a clearer understanding of the time costs, we present a comprehensive time analysis below. Traditional command execution typically follows this workflow: learn command parameters (via ChatGPT or search engines) > locate appropriate file paths through OS lookup mechanisms > input the complete command > and finally obtain results. In contrast, LSFS streamlines this process into a simpler workflow: input natural language commands with file names or semantic keywords > confirm LSFS suggested file operations and obtain results.

In order to quantitatively analyze the time difference between traditional file systems and LSFS, we adopt FS commands and LSFS commands to achieve different file operations shown below. 10 Ph.D. student are invited perform file operations through using Linux commands in traditional file systems and using natural language commands in LSFS, respectively.

The breakdown of time in different steps for the students to operate files correctly is collected and the average time in each step is calculated as below.

LSFS

Traditional FS

While LSFS takes longer for executing file operations due to the inference time of the LLM, the total time it takes for users to successfully perform the file operations is significantly shorter than that of traditional file systems. This is because traditional systems require users much more time to learn through trial-and-error during the "Learn Command" step. In contrast, LSFS simplifies the process through natural language commands, reducing overall time.

We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns:

W1. The motivation is not concretely convincing, especially the first challenge mentioned in Introduction. In the Intro section, the authors mentioned that "For instance, if two files have similar content–such as different versions of the same document–traditional file systems lack the capability to organize or retrieve these files based on their content similarity." Why should the files be organized by the similarity of their content? What are the benefits and what are the practical application scenarios? It would be better to at least add one short discussion or a few examples.

Thank you for raising this question regarding the motivation. We would like to clarify that certain types of documents, such as legal or contract documents, often have multiple versions with very similar content, typically differing by only one or two entries. To be specific, POSIX file system interface only provide basic operations to store and retrieve data, and do not provide any interfaces or semantics based on the contents of the files. However, in LSFS, users can perform content-based retrieval by instruction such as: "Please help me search for the xxx file, which does not include xxx information".

W2. This paper does not point out what key problem they want to solve. Compared to a research paper, it seems more like a technical report.

We would like to highlight our work addresses a fundamental problem observed in agent systems: the lack of a systematic approach to semantic file management and we would like to emphasize the two major research challenges we encounter when designing this system as the following.

• The first challenge is the semantic gap from prompts contain file indentions to actual file operations. Indeed, none of the existing file systems support semantics of data store and retrieval based on the content of files. These traditional file systems rely on exact matches and rigid hierarchical structures, making it difficult to perform operations based on semantic understanding such as 'find documents about machine learning'. To address this challenge, we propose the semantic-based file index structure using vector databases to capture and store file content meanings. Also, we design the semantic parser that accurately translates semantic queries into new designed file operational APIs.
• The second challenge is that simple architecture designs that directly map from language prompts to file system calls can pose security vulnerabilities. For instance, a misinterpreted command could lead to unintended file deletions or destructive overwrites, potentially causing irreversible damage. To address this challenge, we design the layered architecture with high-level file operation API to isolate the direct access to low-level file syscalls. We also design multiple verification mechanisms at both APIs and syscalls to validate operations before execution and design the rollback mechanism that can reverse potentially harmful operations, ensuring operation safety.

W3. The experimental setting is questionable. No baselines and introduction of datasets.

In our experiments, our dataset contains various types of files collected from the Web (e.g., google scholar), such as plain text files ".txt", binary text files ".pdf", and ".doc". Here is our detailed experimental setup:

1. In Experiment 5.1, we validate the effectiveness of our parser using four different LLM backbone. For each API, we created 30 use cases, each with a unique language structure. The test data is provided in the anonymous link.

2. In Experiment 5.2, we evaluated the effectiveness of LSFS in semantic retrieval. Since traditional file systems lack semantic retrieval capabilities, we used LLM alone as the baseline for file retrieval in our evaluation. Specifically, we measured the time and accuracy of executing a semantic file retrieval task by simply feeding file content into LLM to retrieve files and retrieving files using LSFS.

3. In Experiment 5.2, in order to check the robustness of our rollback function when the number of files increases, we tested the rollback time under 5-40 versions respectively, and obtained the robustness of the rollback time.

4. In Experiment 5.3, we compared LSFS with traditional file retrieval methods based on Precision, Recall, and F1 score. The methods include:

   • TFS-search-window: Uses the computer's search window (e.g., MacOS Spotlight) to retrieve both binary and plain text files.
   • TFS-grep: Uses the Linux terminal command grep, which can only retrieve plain text files.
   • TFS-grep*: An enhanced version of TFS-grep that first converts binary files into plain text before using grep to retrieve.

5. In the second part of Experiment 5.3, we compared LSFS and pure prompting to different LLMs as baselines to generate code for creating sharable file links.

W5. Safety insurance mechanisms is pointed out as one contribution, however, there is no description of this mechanism and no experimental comparison between the performance of LSFS with and without the safety insurance mechanisms.

We designed some security mechanisms:

1. We added a process lock to LSFS to prevent consistent reads and writes to the same file
2. We design a user confirmation step: When a user makes a change to a file, the user will be asked to confirm the changed object twice
3. We designed rollback operations: if the user makes a wrong change to the file, they can roll back to the correct version

For first and third mechanisms, the file operation reliability can be improved as long as these two mechanisms are enabled.

For the second mechanism, we conducted a quantitative evaluation of two aspects: the probability of file misoperation and the proportion of risky operations. We compared these metrics with and without the confirmation mechanism enabled.

In the current experiment, we used the retrieval function to locate target files for operations. The table below shows the probability of retrieval errors with and without the confirmation step:

Additionally, we evaluated the proportion of potentially dangerous operations executed (e.g., write, update, or delete) across all file management APIs. The results below demonstrate that the confirmation mechanism in LSFS effectively prevents unintended dangerous operations:

These results highlight that enabling the confirmation mechanism significantly enhances the safety and reliability of file operations in LSFS.

Dear reviewer QPMt,

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. As the deadline for the extended discussion period is nearing, in order for us to have sufficient time to address any additional questions you may have, we kindly encourage you to engage in the ongoing discussion and share any further insights or clarifications you may have.

Thank you very much for your time. We look forward to hearing from you soon.

Best Regards,

All authors