Chain-of-Symbol Prompting for Spatial Relationships in Large Language Models

Thanks for your valuable advice.

For weakness:

1. We manually correct zero-shot CoT's output, which described in Appendix B.2 .

2. Thanks for your kindly mention! We will adjust it in our paper.

3. In addition to spatial tasks, we add experiments on the temporal sequence QA tasks from the BBH benchmark which is shown in the third table of part 1 in general response for all reviewers.

4. Thank you for your encouragement. One intuition we think might be useful is that CoS uses symbols to replace part of natural languages, hence reducing noises brought from redundant natural languages. Neural network might only needs to know the relationship between entity A and other entities to conduct the complex reasoning or planning task. We postulate that removing irrelevant information contained in the natural language can help model to focus on the real task. And using symbols might can reduce potential context information gained from pre-training stage compared with natural languages.

5. We will add more limitation about this aspect in our paper. Although our method uses less natural languages compared with CoT, we actually contains some information of input in CoS ( e.g. objects / entities ), which is easy to trace them back to input descriptions, which can then track the reasoning path.

6. Zero-shot CoT is the most straightforward clue that one could have to use CoS in their own application. While other format can be useful, the search space is huge and it can be spend more time in writing the demonstration fully manually. Also, we tried in our preliminary experiments on human-written CoS and CoT. We did not have promising performance on both CoS and CoT when human writting, and we postulate that LLMs adapts better to the zero-shot CoT format produced by themselves.

Thanks for your valuable advice.

For weakness:

Although we don't propose new "algorithm", we provide insights to use symbols to replace natural languages in demonstration / prompting, which we think would be a promising direction to further explore for future works, for using symbols can wider the representation space in prompting, and reducing potential noises brought from natural languages. What's more, in our additional experiments about open-source LLMs, we find the scaling curve of CoS is steeper than CoT’s and the emergent ability of LLMs for understanding CoS (symbols) when size increases, which is very interesting. We also add different tasks of scaling curve in the latest uploaded version of the paper in figure 4, shows the general trend across tasks.

For question 1:

In addition to spatial tasks, we add experiments on the temporal sequence tasks from the BBH benchmark, which is very similar to the situation you mentioned, and results is reported in the response for all reviewers. The prompting demonstration and examples are added in latest uploaded version of our paper in the table 16.

For question 2:

We use CoS for two reasons:

Not only using natural languages but also using symbols can make the representation space wider in prompting.

CoS uses symbols to replace parts of natural languages, hence reducing noises brought from redundant natural languages. One assumption is that words of natural languages might have more context information during pre-training, which might bring potential additional information, but using symbols can avoid these, by connecting objects without potential additional information, and make prompting more clear and structural.

Thanks for your valuable reply and advice, we add open-source experiments in the response for all reviewers.

For weakness:

Point 1:

We will add a clear definition of universal guideline to convert tasks into condensed symbols in Section 3 of the paper.

We first define our task in a more general perspective:

We can formulate the tasks which our method is suitable as a set of tasks which has sequential and limited number of relationships, and the relationships can be spatial, and also can be other kinds such as temporal (which we extend as new experiments in the response for all reviewers).

Then we define the procedure of our conversion from natural lanugages of CoT to our CoS:

For converting natural languages into symbols, we can first use a recognizer to recognize objects and entities in the natural language descriptions, e.g. using human, NER system or even rule-based codes to automatically recognise objects in the sequential relationships described by natural languages, and then connect them with each other only using random symbols ( e.g. '/', '-', etc). By doing this, we actually use symbols to replace original relationships which connect objects together. For simplicity, we can pre-defined the set of symbols, and the mapping between each kinds of symbols to each kinds of relationships.

What's more, although we only verified on specific task examples, the idea itself can provide insights for future research: (e.g. customizing the conversion process, customizing mapping between symbols and relationships, even customizing the way of using symbols in in-context learning), what important is, we provide insights to use symbols to replace natural languages in demonstration / prompting, which would be a promising direction to further explore for future works, for using symbols can wider the representation space in prompting, and reducing potential noises brought from natural languages.

Point 2:

Based on guideline we give in the point 1 and in the section 3 of the paper, we can easily transform any sequential natural language exemplars which meet the conditions to chain-of-symbol exemplars. The guideline is suitable for all tasks which meet the conditions, we have proved its effectiveness in spatial relationships and temporal sequence reasoning tasks as examples, and future work can explore more tasks.

Point 3:

For open-source models, the results and analyses are concluded in part 1 of general response for all reviewers. Figure 4 in our latest uploaded version of paper gives a clear results about scaling curves of CoS compared with CoT, it shows CoS has more steep curve for all tasks, which indicates it is a general ability that LLMs can understand abstract symbols better when size increases.

For questions:

Point 1:

We conclude this point in the second part of general response for all reviewers, specifically, we decompose this into two dimension: task domain and task complexity.

For task domain, inspired from Reviewer c5jB, we use similar scenarios in BBH benchmark, called "temporal sequence", it is a reasoning task which provided a sequence of time slots and related scenarios, and model should answer which time is available based on given descriptions. And in the above second table, we can see that using CoS can outperform CoT significantly (from 70 to 93.) We also conduct experiments on GPT-3.5-Turbo, and shows CoS can outperform CoT from 66.5 to 76.2

For tasks complexity:

We conduct control experiments where we control the complexity of the task by setting the number of bricks in brick-1d scenarios. When the number of bricks increases, which means the complexity of the task increases, the performance of CoS can outperform CoT. It verify the robustness of CoS when task becoming hard.

For whether there is an automatical way to convert or not, see the point 1 for weakness above.

Point 2:

For other methods, for example program-based CoT, can you be more specific? We only find this related work https://openreview.net/pdf?id=YfZ4ZPt8zd, which uses an external program executor to help calculate, but to be honest it is not a fair comparison with our CoS method which only does the inference by LLMs on their own, but we can list these kinds of works in our related work section.

Point 3:

The computation required to convert symbols is down to 10ms through Python program based on given CoT exemplars. For those tasks which do not have manual annotated symbols, we can follow the procedure described in the part 1 of weakness above. For a broader range of tasks, just as the description in part 2 of general response for all reviewers, we explore temporal sequence tasks and verified the effectiveness of our method.

Dear reviewer tveC,

We greatly appreciate your valuable feedback. We conduct additional experiments and write a detailed shared response and specific response for your review.

As you may know, unlike previous years, the discussion period this year can only last until today (only 3 hours left).

If you are satisfied with our response, please consider updating your score. If you need any clarification, please feel free to discuss with us.

Best,

Authors

Thanks for your reply and advice, we follow your advice and complete several new experiments, which described in the general response for all reviewers. Below is some response for weakness and question.

For weakness 1:

You can see the part 1 of general response for all reviewers, we add the experiments of open-source LLMs, llama-2 with different sizes. And you can also see the figure 4 in latest uploaded version of our paper very clearly about the more steep scaling curve of our method across different tasks compared with CoT. It shows the effectiveness of our method compared with CoT and the enmergent ability of understanding CoS.

For weakness 2:

Thanks for pointing out our misleading part, We revised our claim in the revision version.

For the concern of marginal improvements, we add further experiments about using longer description instances (e.g. from 6 to 14 bricks) and find that CoS can gain more improvements compared with CoT in these scenarios compared with more simple scenarios, which is reported in part 2 of general response for all reviewers.

For "the impressive gains are more about the simplicity of the Brick World dataset", actually the "shuffle both" results of brick 1d shows it is not a simple task as it looks like, where CoT only gains 43.0 accuracy. This actually shows that LLMs do not have good spatial understanding capabilities, because they cannot do spatial reasoning tasks which human think is easy well.

For weakness 3:

We add experiments on the temporal sequence tasks from the BBH benchmark [1]. This is another task which is not related with spatial relationships, and which also contains a more real-life situation. We report the results in table 2, part 1 of general response for all reviewers, and also tested on GPT-3.5-Turbo, and shows CoS can outperform CoT from 66.5 to 76.2.

"It seems that the success of this method, referred to as CoS, might be limited to simpler, more straightforward spatial tasks." We use complexity control experiment and task of temporal sequence of BBH benchmark to shows the contribution of our proposed method, and we also want to describe our method in a more general way:

We will add this into Section 3 in our paper: we can formulate the tasks which our method is suitable as a set of tasks which has sequential and limited number of relationships, and the relationships can be spatial, and also can be other kinds such as temporal.

For converting natural languages into symbols, we can use human, NER system or even rule-based codes to automatically recognise objects in the sequential relationships described by natural languages, and then connect them with each other using random symbols ( e.g. '/', '-', etc). By doing this, we actually use symbols to replace original relationships which connect objects together.

Generally, we want to provide insights about using symbols in demonstration of the prompting, and we provide a particular way to use symbols to make LLMs to be more effective and efficient when handle sequential tasks.

For question:

We only have limited access of GPT-4 API currently, we will add the related results when we can access GPT-4 API in our future version.

[1] Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. ACL 2023 Findings

There are only two days remaining for the author-reviewer discussion (November 22nd). Please read through the authors' response to your review and comment on the extent to which it addresses your questions/concerns.