Evolved LLM Schemas for Mid Vision Feedback

Thank you for your thoughtful and constructive feedback. We structure our response to you based on the different points you make about improving the work. The points you make in the "Weaknesses" and "Questions" section are fully addressed within the sections below. We have made our paper stronger as a result of your feedback, and have uploaded it.

The approach for MVF is hard to follow Thank you for your feedback. We add to the Appendix a more in depth description of the workings of MVF. In it we clarify "affine transformation" in text to make its meaning clear (see lines 53, 206, and 291).

Some format issues are found Thank you for catching these! Most of this feedback was used to improve the draft, with the exception of the last. The formatting of the conference name in each case is taken from google scholar, and we use this formatting so that readers may easily use google scholar in finding these papers.

Example Schemas Thank you for pointing this out. We agree it would be helpful for future readers of the work to see some example schemas to understand the representation. However, schemas are discovered over the entire dataset - not over individual images. As such, we display schemas over the entire CIFAR dataset rather than associated with individual images. Please see Section A3 in the Appendix.

Novelty Clarification

To our knowledge we are the first to integrate EvoPrompt and MVF, and more generally, the first to employ high level schema modeling in biasing internal network features in the manner of ELF. This combination is not only technically novel, but conceptually novel: the effect is a bidirectional integration of internal visual representations in CV architectures with semantic contextual modeling from LLMs. As demonstrated through our experiments, the incorporation of EvoPrompt and MVF working jointly improves performance of base networks.

Ablations Thank you for pointing this out, the ablations referred to were Figures 6 and 7. We make note of this in Section 4 (see 353 - 354).

SOTA Models The SOTA models over CIFAR100, Caltech101, and ImageNet are complex ensembles of models, with hyperparameters and training details selected carefully. The usage of feedback within these architectures would warrant extensive experimentation. The models we employ as well as the additional model added in the newer version of the paper make for a total of 6 models over which we demonstrate the utility of ELF. Due to the architecture complexity of the present state-of-the-art models on these datasets, integration into these architectures is beyond the scope of the present work. While experiments over models beyond the 6 already evaluate would strengthen the evaluation even further, we believe that the present experiments present a solid evaluation of the principle of ELF.

Thank you for your thoughtful and constructive feedback. We structure our response to you based on the different points you make about improving the work. The points you make in the "Weaknesses" and "Questions" section are fully addressed within the sections below. We have made our paper stronger as a result of your feedback, and have uploaded it.

EvoPrompt and Mid Vision Feedback (MVF) Thank you for pointing out confusion over the workings of EvoPrompt and MVF - we add to the Appendix summaries of these methods, providing an overview which is more comprehensive than there is space for within the paper. The implementation details of EvoPrompt and MVF are not part of the contribution of ELF.

In the Appendix we make sure to include the equations adopted in the original works (EvoPrompting and MVF) for purposes of clarity and specificity. Please see Appendix sections A4 and A5.

Experiments Thank you for pointing out the missing baseline with respect to the original MVF work - we extend our experimentation, choosing the single context that produces the highest accuracy using the mechanism in the original Mid-Vision Feedback work - see the updates to Tables 1 and 2.

Experimental Details Thank you for pointing out the missing experimental details. We have included missing experimental details, including that which you pointed out was missing. The input resolution during the evolutionary search is $32\times32$, but when transferring the schemas to new datasets, the ViT and ResNet50 models ingest images of resolution $224\times224$.

Schema Optimization Figure 7 displays the benefits of evolutionary search - the seeds produce accuracies hovering around $60\%$, and the accuracies produced by the schemas derived from evolutionary search display sizable increases as the evolutionary search progresses. We hope the new qualitative Figure in the Appendix further provides further illumination (please see A2).

Questions:

Addressing your question, we modify Figure 3, and alter its caption, for clarity. And as per your suggestion, we alter the presentation in Tables 1 and 2, removing the margins columns.

Thank you for your thoughtful and constructive feedback. We structure our response to you based on the different points you make about improving the work. The points you make in the "Weaknesses" and "Questions" section are fully addressed within the sections below. We have made our paper stronger as a result of your feedback, and have uploaded it.

Abstract Thank you for pointing out issues of clarity in the abstract - we update the phrasing to make it clear what these terms are referring to.

Introduction Thank you for pointing out the confusion readers might encounter with the Introduction. We briefly describe the task of image classification approached in this work, and the lack of adoption of context modeling/schemas in image classification. See Lines 63 - 71.

Confusing Figures We rework Figure 3, and the caption of Figure 3, to make the process flow clearer, numbering steps and clarifying associated descriptions. Because Figure 3 involves evolutionary search, which is an iterative rather than single pass process, Figure 3 involves cycles. These cycles may make the beginning and end less apparent as each component is revisited $N$ times in the search.

Technical implementation of feedback Thank you for pointing out the lack of clarity. As the details to Figure 5 and the mechanism of feedback require an in-depth understanding of the original MVF paper, we point the reader to the Appendix (A4), where we describe the feedback mechanism in-depth.

Thank you for your questions on Figure 5. These are valid questions, though they are beyond the scope of Figure 5. Figure 5 seeks to illustrate the impact of applying affine transformations over feature vector representations, but to do so without addressing where these affine transformations come from, or how they are trained. This is described in the original MVF paper, and we add a more extensive description of this training process in the Appendix (A4).

Flop Analysis There is likely a misunderstanding here, as the forward pass of the vision networks during training/inference requires only two forward passes of the network - the first forward pass to predict context, and the second forward pass to do classification. The details of this are contained within the Appendix.

While the question of computational complexity in the training process is an interesting one, computationally efficient training is not the objective of ELF, nor in general the objective of evolutionary search algorithms. ELF seeks to maximize test time performance, while not adding computational burden during test time.

Injection Level The optimal injection level for the feedback mechanism was taken directly from the original feedback paper, where the penultimate layer was chosen across all architectures. As the feedback mechanism used in ELF is identical to that of MVF, we adopt this injection level and refer the reader to MVF for the accuracies over different injection levels.

Experiments on larger, more modern models Present experiments across 5 different architectures demonstrate consistent benefit to ELF. These include a range of architectures, including both CNN and transformer architectures. We include one more set of results adopting a state-of-the-art, ViT-L network pre-trained over ImageNet. Please see Table 2. We note however that both ResNet50 and ViT-Bx16 networks are already pre-trained, and make mentions to this in the Experiments section.

Exact Prompt Thank you for this question - we include the prompt in exactness in the Appendix (please see A6). In the prompt, we do not name the dataset, but rather only the list classes constituting the dataset. This prevents the network from dataset-specific bias.

We also observe that final ELF performance is not highly sensitive to initial context seed set (see response to Reviewer 1), and as such a priori familiarity with the dataset in question would not be required of GPT-4. Also consider the generic nature of produced contexts: context seeds sets are discernable from the object list of a dataset, and are not specific to the dataset.

Other Vision Tasks Feedback has been shown to help in the task of video classification, and we believe can benefit networks trained over the task of image segmentation. As such, there is a strong case for the application of ELF in both of these tasks. We leave this to future work.

Pre-training All networks are pre-trained over ImageNet. We make a note of this in the Experiments section.

Thank you for your thoughtful and constructive feedback. We structure our response to you based on the different points you make about improving the work. The points you make in the "Weaknesses" and "Questions" section are fully addressed within the sections below. We have made our paper stronger as a result of your feedback, and have uploaded it.

Context limitations - Rather than the context ontology used in this work being limited, our ontology includes $26$ different contexts. We include the full list in the updated Appendix section, in Section A1.

Transferability - The transferability of schemas across datasets is one advantage of ELF. As schemas are domain knowledge representations, they are transferable to the extent that domains are shared across datasets. Many of the contexts selected by ELF are general in nature (e.g., animate/inanimate, nature/urban, as you point out), and as such are generalizable across a wide range of datasets. However, contexts produced for one dataset will not generalize to a different dataset with no domain overlap - for example, schemas produced for a dataset of birds will not generalize to a dataset of manufacturing settings.

The present ELF contexts have high intraclass variability (e.g., all inanimate objects is a class with high intra-class variability), which allows it to generalize across datasets. Datasets which share no structure cannot be generalized across. Note that the object ontologies of the datasets evaluated differ.

A more extensive consideration across more than the 3 datasets evaluated could provide some benefit, though we believe that the demonstrated generalizability of ELF models across these datasets effectively demonstrates that the principle of ELF is solid.

Comparison to other context modulation works - Thank you for pointing out this missing baseline. We incorporate the context modulation approach from MVF, choosing the single context in the ontology that produces the highest accuracy. Please refer to the baseline MVF in Tables 1 and 2.

Failure cases - Thank you for pointing this out. We agree it would be helpful for future readers of the work to see some example schemas to understand the representation. Please see the newly added figure in the Supplementary Materials (A2), where we also address your first question. Typically, more complex schemas perform better within ELF while being less interpretable than the simpler schemas.

Industry Applicability - In the present evaluation we employed datasets with conventional ontologies, with categories familiar to GPT-4. In a highly specialized setting involving specialized manufacturing equipment with which GPT-4 has minimal understanding one challenge would be on seed context generation. One work around to this could be hand engineering seed contexts, or employing an LLM model which was trained over domain specific text sources. However - as per the next response - we observe that final performance is not highly sensitive to the initial seed context set, so the adaptation to highly specialized domains may require little effort in adjustment. We add this content to a separate "Limitations" subsection within the Appendix (A3).

Different Seeds, different LLMs - We found the question of seed selection interesting to explore as well. We observed no noticeable impact on the final performance whether the seeds were handwritten or the seeds were automatically generated by GPT4. Furthermore, selecting seeds known to perform well or perform poorly had no noticeable impact on the final classification performance after evolutionary search. We add a short note to the caption of Figure 6 mentioning this. As for the question of whether different LLMs perform better, please see Figure 6 where we compare GPT4, GPT3.5-Turbo, and GPT4-Mini. The choice of LLM makes a pronounced difference in final classification results.