Simple mechanisms for representing, indexing and manipulating concepts

We thank the reviewer for their time and effort in providing feedback on our work.

Comments on examples or experiments: In section 4, we show how to obtain the signature of a concept using circles as an example. We also show that by combining signatures of multiple lower concepts (i.e., signatures of concentric circles with different radius), one can obtain the signatures of a higher-level concept (i.e., a general concept of concentric circles). We also show how simple operations on top of signatures yield interesting properties (e.g., rotation and translation invariant signatures of the manifold). If the reviewer can point to any particular part of the paper that would improve with an example, we will try our best to provide such an example.

Details about the proposed architecture: We have updated Section 3 to include more information about our proposed architecture and also updated Figure 2 to include more details. Our analysis points to an architecture with a “concept discovery unit” (which is similar to a traditional Transformer but likely smaller as is not used to remember concepts) and a separate “concept dictionary” which remembers all the concepts found in the training data. This separation helps to scale up the memory and number of concepts learned efficiently. Further, the nullspace concept signatures could add to the level of interpretability of the concepts present in an input.

Connection to Transformer: We have added a figure (Figure 3) to explain the connection of our proposed learning architecture to transformer architectures. On a high level, a random feedforward network can contain the signature $S(x) = \phi(x) \phi(x)^\top$ for each token and attention can be thought of as combining tokens from the same concept together to obtain $M(X)$ and the subsequent feedforward network can be used to obtain $T(X)$ using $M(X)$ because $T(X)$ is the null space of $M(X)$ and it can be obtained by taking by repeated multiplication of $(I-M(X))$. We have added this discussion to the updated paper.

We hope that our response resolves the concerns of the reviewer and we are happy to discuss more to increase the confidence of the reviewer in our work.

We thank the reviewer for their time and effort in providing feedback on our work. We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

Regarding the question about polynomial manifolds, the reviewer is right that this is an assumption that constraints on the inputs. However, the embedding mapping is meant to capture them and allows for wider applicability.

Regarding the question about polynomial manifolds, the reviewer is right that we are constraining vector representations of different instances of a concept to change smoothly and lie in a manifold however, we believe that even if this is not true in the input layer as one computes signatures up the layers hierarchically one gets smoother representations . Although we have argued this concretely only for simple concepts such as circles, rectangles, and stick figures we believe this should be true more generally.

We hope that our response resolves the concerns of the reviewer and we are happy to discuss more to clarify any further concerns.

We thank the reviewer for their time and effort in providing feedback on our work.

Answers to weaknesses

As the reviewer points out, our formalism does suggest a modified transform architecture that includes a separate concept storage unit (dictionary of concepts) and a concept discovery unit, which is much smaller than today’s giant transformers. See Figure 2 in the modified pdf. Thus, our theory formalizes the importance of retrieval-augmented transformer architectures.

Proposition 1.1 is to describe the main results before going into the detailed statements presented later in the paper. We can change the name ‘Proposition 1.1’ to “Informal summary of results”.

Answers to questions:

1. There is no restriction of parameters in the definition, but typically we would have $d < m$ (feature expansion by the kernel).
2. We note that the kernel signature is defined specific to a dataset and the choice of the kernel phi. For MNIST, one could view each image as a point cloud where each point represents (pixel-coordinates, pixel-value). You could now compute the kernel signature for each image $M(x) = \mathbb{E}[ \phi(x) \phi(x)^\top ]$ where expectation over a group of related points that attend to each other. As explained in Section 3.1, the attention will cause points from the same continuous curve to attend to each other. If the image had only one curve (e.g., an image of number “0”), then this would result in a signature for that curve. An image containing multiple curves (such as the image of number “7”) would result in two signatures at the second layer. The third layer will group these two curves into one signature (see section 3.2) and this signature can be used to identify similar images.
3. The notation $\mathtt{S}^{d-1}$ denotes the unit-sphere in $d$-dimensions, and we will add this clarification.
4. The learning method uses a previously computed representation (such as $\phi(x)$) and applies attention on $\phi(x)$ based on cosine similarity (see figure 3 for more details where the attention is actually over $S(x)$ which is $\phi(x) \phi(x)^\top$. Thus, $S(x).S(y) = (\phi(x).\phi(y))^2 \ )$. As the reviewer points out, cosine similarity cannot distinguish manifolds with the same orientation but different distances but since we are using the kernel transform, they can be distinguished by a suitable kernel (for example, if the distance to a reference point is included as one of the features in the kernel).
5. We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

We hope that our response resolves the concerns of the reviewer and we are happy to discuss more to clarify any further concerns.

We thank the reviewer for their time and effort in providing feedback on our work. We agree with the reviewer that the present paper lacks experimental work. The paper aimed to take a first-principles approach to the challenging problem of formalizing the recognition of concepts and deduce the implications of such a formalism. While there are several empirical investigations of such questions for existing architectures and methods, the formalism of concepts is quite poorly understood, and it would help to develop theoretical formalism first without being bogged down by the noise in data, variations in architectures, training methods, and datasets. Designing extensive experiments and developing the framework for real-world datasets will be a substantial body of work by itself, just as developing the formalism here was.

We hope our response resolves the reviewer's concerns, and we are happy to discuss further to clarify any further concerns.