Transformers Perform In-Context Learning through Neural Networks

Thank you for your valuable comments and suggestions. We provide our response below.

The novelty compared to Bai et al, 2023

Apart from generalizing the 2-layer NN setting to the n-layer setting, we actually used a different approximation theorem to achieve a better error bound for our results. It leads to the decrease of number of heads and hidden dimension from $O(\log\frac{1}{\epsilon}\epsilon^{-2})$ in [1] to $O(\epsilon^{-2})$ in our result.

Thank you for your valuable comments and suggestions. We provide our response to each question below.

Question 1. Experimental results.

So far many works ([1], [2], [3]) have theoretically proven the advantage of deep networks over shallow networks, but there lacks experimental results. Empirical validation still needs further investigation.

Question 2. Assumption 2

The domain $\mathcal{W}$ of $w$ is a compact set in $\mathbb{R}^d$, meaning its bounded and closed. Here $d$ is the dimension of $w$. The existence of such an MLP layer requires some construction, but it is reasonable to assume the existence since an MLP layer projects the input into another space.

[1] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pages 907–940. PMLR, 2016.

[2] Shiyu Liang and Rayadurgam Srikant. Why deep neural networks for function approximation? arXiv preprint arXiv:1610.04161, 2016.

[3] Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In International conference on machine learning, pages 2979–2987. PMLR, 2017.

Thank you for your valuable comments and suggestions. We provide our response to each question below.

Question1. The definition of $W_{loc}^{m,\infty}(\mathbb{R})$

$W^{m,\infty}(\mathbb{R})$ denotes the Sobolev space that is a subset of the $L^\infty$ space. For any $f\in W^{m,\infty}$ we have $f$ is continuously differentiable up to order $m$, and these derivatives have finite $L^\infty$ norm. Here $W_{loc}^{m,\infty}(\mathbb{R})$ is equivalent to $W^{m,\infty}(\Omega)$, where $\Omega\subset \mathbb{R}$ is a bounded domain.

Question 2. Meaning of assumption 2

This assumption follows Bai et al, 2023 in Theorem G.1. Since $\mathcal{W}$ is a bounded domain, we assume that an MLP layer can project $w$ into its domain for simplicity, and whether there's a concrete example remains requires some construction, but it is reasonable to assume that such a layer exists due to the essence of the MLP layer, which projects the input into another space.

Question 3. Functions having finite Barron norm

We would like to refer the reviewer to section IX in [1] where Barron discussed functions that have finite Barron norm. These functions include sigmoidal functions, radial functions, linear functions and polynomials, and most importantly, functions of sufficiently high order, with partial derivatives of $f(x)$ of order $s=[d/2]+2$ continuous on $\mathbb{R}^d$. So the commonly used $l_2$ loss function clearly falls into this category.

Question 4. Results in section 4

First we know that a transformer can learn neural networks in-context, meaning given the input, a transformer can give the output of the NN, and what we mean by "learn the neural network" is based on this fact. So our takeaway message is "When a transformer learns a function in-context, it actually learns the neural network that approximates the function in-context".

[1] Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945, 1993.