ScenicNL: Generating Probabilistic Scenario Programs from Natural Language

Did you think of an evaluation of the final generation?

We refer the reviewer to Section 4, where we did perform a full evaluation. As part of our evaluations, we had 3 people familiar with Scenic read the output programs to judge them for their Accuracy, Relevance, and Expressiveness. Details of the rubric used will be provided in the appendix. This is one of the main benefits of using Scenic as an abstraction over simulators i.e. Scenic eliminates the need for sitting through hours of video to judge a simulation. Consider the example of an object smaller than a car blocking the path of that car. If the camera angle of the simulator is set to follow the vehicle, no one would ever know what object is blocking the vehicle. However, by reading the program, one would immediately realize there is an object located in the same lane as the vehicle.

In Section 3.5, we experimented with using a Vision-Language Model to determine if such models could automate part of the human evaluations. However, as mentioned, it is left as future work as there are several problems which limit the VLMs from replacing our human evaluators: (i) how many and what camera angles are needed to capture all objects? (ii) when the VLM fails to understand spatial relationships between objects, how do we proceed? (iii) should the feedback be incorporated into the natural language reasoning of the LLM judges or directly into the programs?

Will you release your code...

Yes, by camera ready

How did you create the example scenic programs from Fig. 1?

The example programs used in Figure 1 were sourced from the publicly available programs in Scenic’s Github. Specifically those using CARLA as a backend simulator: https://github.com/BerkeleyLearnVerify/Scenic/tree/main/examples/carla

The authors contribution is sometimes difficult to evaluate...

We refer the reviewer to the several existing works that we cited which have either (i) used a similar data source and performed reconstructions in simulation by paying many crash reconstructions experts to do it by hand (Scanlon et al. with Waymo) or (ii) solved a limited set of scenarios which were not real-world crash scenarios but rather simple driving examples e.g. TARGET by Deng et al. Still these methods were done by using a template based approach e.g. ADEPT by Wang et al.

Do you plan to release your code and the generated programs...

We will release our code by camera ready.

The paper is more like a technical report...

Our baseline evaluations demonstrate that LLMs cannot generalize to writing code for domain specific languages. However, as shown in our work, the combination of filtering, prompting, and reasoning can help guide an LLM towards writing code in new settings. The current space of ways one can use an LLM is huge and continuously growing. Our work offers one successful method in a space full of many unsuccessful paths.

Concerning limited research interest, we kindly refer the reviewer to the introduction which cites several works from both industry & academia that have attempted to solve the same problem but in a limited manner.

Though the paper tries to claim that their methods are not unique...

One of our motivations was to study simulation of multi-agent systems in physical environments. This motivation is shared by industry, e.g., OpenAI’s SORA is their proposal for “general purpose simulators of the physical world” (https://openai.com/index/video-generation-models-as-world-simulators/). Unlike OpenAI, we wanted to an interpretable, controllable representation with precise semantics for simulation generation and control. Scenic was a natural choice that fit this, as evident by the related works that have also tried to generate simulations using Scenic. As per Scenic’s documentation, it supports several simulators apart from CARLA and is used in other domains like Aviation & Robotics. While these other domains are interesting, they do not offer a dataset of recorded failures. As mentioned in the introduction, several regulatory bodies record these interesting failure cases for autonomous vehicles. Finally, as mentioned in Section 3.3, we keep all reasoning and soliciting of information in natural language. The only portion which includes Scenic specific statements are the Few-Shot & Constrained Decoding prompts to help generate syntactically correct code.

As an application oriented paper...

Simulation based testing is a popular evaluation method for industry leaders such as Waymo. One potential use is for the companies whose accidents we replicated to retest their simulated vehicles using our programs or as an evaluation benchmark. Another possibility as mentioned in the conclusion, is to leverage the simulations in the training of RL based agents (also an area where the Scenic has been used).

Why is expressiveness interesting to track...

Expressiveness is a property of probabilistic programs which is more heavily used than in imperative programs. For example, a police report may indicate that a vehicle was traveling at “approximately 35 mph”. A low expressiveness would be from a program that just assigns that speed directly. A high score would be from a program that assigns the speed as a distribution (around 35 mph). This is valuable downstream because each sampled program would result in a different vehicle speed which could lead to different behaviors and different findings of how the autonomous driving systems react.

Accuracy and Relevance are also human evaluated measures recorded in our work. The full rubric of the proposed 3 measures will be provided in the Appendix.

There are few details of the evaluation setup. ...

3 people familiar with Scenic randomly evaluated a subset of the outputs per experiment. Please let us know if further questions remain

The RAG/RAG+HyDE performance is very high compared to the rest. ...

The examples used are the handful publicly available in the Scenic Github. These example programs are not related to autonomous vehicle accidents. In fact, to the best of our knowledge there is no current work that leverages the crash reports of the California DMV, therefore, all scenarios used were not previously seen. Could you kindly clarify the former question? The compilation rate of RAG was 1% and RAG+HyDE was 7% as detailed in Table 1, which in fact was not the highest of the baseline methods.

Have you tried to provide Scenic documentation in a prompt? ...

We tried but did not run a full baseline evaluation on due to poor performance. All Scenic documentation is on Github. We took the full documentation (64,265 tokens using the llama tokenizer), chunked it, and provided it into a Pinecone Vector DB. We then tried using RAG only with this database as well as a combined approach of RAG from this database AND RAG with a single Scenic program retrieved (from all available Scenic programs even those using different simulators in different domains for a total of 73,825 tokens) but found that in either case, performance was poor. This is most likely due to the documentation explaining how things are implemented under the hood (no pun intended) rather than offering examples and explaining proper usage of Scenic features and syntax.