Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions

We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.

The paper might benefit from offering more intuitions or informal explanations for its results. Overall, it is somewhat challenging to read for those not deeply engaged with the formal or theoretical aspects of machine learning.

Following the reviewer’s comment, we will include the following informal explanations describing our intuition behind our proof at the beginning of Section 4.

“We construct the target four-layer network as a linear combination of indicator functions of points (Lemma 4.1). Given $z$ in the domain, our construction of the indicator function of $z$ consists of three parts. In the first part, we create an injective layer consisting of an affine transformation followed by an activation function based on the distinguishability of the activation function. Due to the injectivity, all inputs in the domain have distinct output vectors after passing this part. Here, we use $y=(y_1,\dots,y_k)$ to denote the output of the first part of $z$. In the second part, we construct $2n$ binary step functions $f_{1,1},f_{1,2}, \dots,f_{n,1}, f_{n,2}$ where $f_{i,1}(x)=(C_1-C_0)1[x\ge y_i]+C_0$ and $f_{i, 2}(x)=(C_1-C_0)1[x\le y_i]+C_0$. Namely, all $f_{i,j}$ are $C_1$ if and only if the input to the network is $z$. To implement such $f_{i,j}$, we exploit the round-off error in floating-point operations and show that we can increase the gap between arbitrary two numbers by sequentially adding the same floating-point numbers to those two numbers (see Lemma 4.6). In the third part, we construct a function that outputs $C_1$ or $C_2$ if all $f_{i,j}$ are $C_1$ and outputs $C_0$ otherwise. We then apply the activation function after this function so that our indicator function is either zero (i.e., $\sigma(C_0)$) if the input is $z$ or some non-zero value (i.e., $\sigma(C_1)$ or $\sigma(C_2)$) otherwise.”

The abstract is slightly unclear because it does not explicitly specify what "almost all" pertains to. While this level of ambiguity might be acceptable for the title, I believe that the abstract should offer more clarification in this regard for better comprehension.

We thank the reviewer for pointing out a confusing expression. Following the reviewer’s comment, we will change the last sentence in the abstract from

“Our result shows that with practical activation functions, floating-point neural networks can represent almost all floating-point functions.”

to

“Our result shows that with practical activation functions, floating-point neural networks can represent floating-point functions from a wide domain to all finite or infinite floats, where the domain is all finite floats for the activations sigmoid and tanh, and it is all floats of magnitude less than $1/8$ times the largest float for the activations ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.”

As I am not familiar with the work directly related, can you make clear how the arguments of Park et al. differ from yours? I assume they rely on similar "constructive arguments"?

The network construction of Park et al. (2024) and that of our work are both based on a linear combination of indicator functions, as in our Lemma 4.1. However, Park et al.’s construction of indicator functions is specialized to ReLU and binary step activation functions (since they consider only these two activations), whereas our construction covers general activation functions satisfying Condition 1.

• For example, in (Park et al., 2024), they use the vanilla binary step function as an indicator function; for ReLU, they construct an indicator function using the following intuition: $1[x\ge z] \approx \text{ReLU}\left(\frac1{z-z^-}(x-z^-)\right) - \text{ReLU}\left(\frac1{z-z^-}(x-z)\right)$.

• On the other hand, our construction of indicator functions is completely different: we mainly use the properties of a floating-point summation and the round-off errors from floating-point additions (Lemma 4.6).

We would be happy to clarify any concerns or answer any questions that may come up during the discussion period.

We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.

Lack of empirical validation. While the theoretical results are robust, the absence of experimental results or case studies makes it difficult to assess the practical implications and real-world performance of the proposed methods.

Since our universal approximator construction consists of an affine combination of indicator functions (Section 4), following the reviewer’s comment, we have empirically verified our indicator function constructions (Lemma 4.1) using GELU networks and 16-bit floating-point arithmetic (float16). Specifically, we constructed indicator functions $f_z(x) := 1[x=z]$ of points $z \in [-1,1]\_{\mathbb F}$ (whose domain is $[-1,1]\_{\mathbb F}$), where $\mathbb F$ denotes the set of 16-bit floats. We sampled $120$ random $z$ from $[-1,1]\_{\mathbb F}$ (i.e., $120$ indicator functions), and constructed the indicator functions $f_z$ using GELU networks. For each $z$, we checked whether our construction generates the same output as in $1[x=z]$ for $1000$ random inputs $x$ drawn from $[-1,1]\_{\mathbb F}$. According to our experimental results, the outputs of our constructions coincided with those of the true indicator functions, confirming that our construction is correct.

The code is available at the following anonymous link:

https://anonymous.4open.science/r/floating-points-indicators-A112/

To run the code, use the following command (trials: the number of indicator constructions, verification_steps: the number of verification steps for each constructed indicator) with NumPy version 1.24.0:

python indicator_test.py --activation gelu --trials 120 --verification_steps 1000

While the paper covers a wide range of activation functions, it does not explore the potential trade-offs or practical challenges associated with implementing these functions in real-world systems, such as computational efficiency or hardware constraints.

Theorem 3.4 (our main result) holds for all floating-point activation functions that satisfy Condition 1 and a distinguishability condition. Corollaries 3.9 and 3.11 show that the following are examples of such activation functions: the correctly-rounded versions of many popular real-valued activation functions, such as id (the identity function), sin, tanh, ReLU, ELU, SeLU, GELU, Swish, Mish, and Sigmoid. We believe there would be no practical challenges in implementing these functions in real-world systems:

• The following functions are already implemented in real-world systems and widely used: $\lceil \text{id} \rfloor$ and $\lceil \text{ReLU} \rfloor$ are available in deep learning frameworks (e.g., PyTorch, TensorFlow, JAX) as torch.nn.Identity()(x), torch.nn.ReLU()(x), etc; $\lceil \text{sin} \rfloor$ and $\lceil \text{tanh} \rfloor$ are available in correctly-rounded math libraries (e.g., RLibm, CORE-MATH, CRLibm) as sin(x) and tanh(x).

• To our knowledge, the other functions listed above (i.e., the correctly-rounded versions of ELU, SeLU, etc.) are not yet implemented in real-world systems. Nevertheless, we believe these functions will also be implemented in the near future, especially given recent active research on correctly-rounded math libraries in various areas such as computer arithmetic (e.g., [Sibidanov+ 2022]) and programming languages (e.g., [Lim+ 2021]).

[Sibidanov+ 2022] The CORE-MATH Project, ARITH 2022.
[Lim+ 2021] High performance correctly rounded math libraries for 32-bit floating point representations, PLDI 2021.

We would be happy to clarify any concerns or answer any questions that may come up during the discussion period.

We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.

It is very hard to read. Sometimes, I have a feeling symbols are not well defined and explained. For example in Definition 3.5, I do not know what $\eta^+$ and $\eta^-$ is and where it come from.

For better readability, we will add a table describing notations and definitions in the Appendix of the final version. $\eta^+$ and $\eta^-$ denote the smallest float that is larger than $\eta$ and the largest float that is smaller than $\eta$, respectively. Although we defined these notations in Section 2.1, we will recall their definitions after Definition 3.5. We will also recall seldom-used notations in the final draft.

The supplementary material is absolutely necessary for understanding. In my opinion, the authors could help the reader by repeating the relevant definitions from the main text, such that the reader is spared from going back and forth.

Following the reviewer’s comment, we will make the Appendix self-contained in the final version, by redefining notations and definitions at the beginning of the Appendix along with a table summarizing these notations and definitions.

We would be happy to clarify any concerns or answer any questions that may come up during the discussion period.

We appreciate the reviewer for their valuable comments. Our response is below.

I don't see how the generalizations of this paper are helpful for understanding neural networks in real-world applications.

Classical universal approximation theorems establish that sufficiently wide neural networks can approximate any continuous function. Although achieving universal approximation requires an impractically large number of parameters, these results remain important: as Reviewer ZVKU points out, they help researchers understand the theoretical limits and fundamental properties of the tools they use.

Our work extends these results by considering neural networks with general activation functions operating under floating-point arithmetic. In contrast, recent related works consider narrower or less practical settings: [Park+ 2024] focuses only on ReLU and Step activations, while [Hwang+ 2024] analyzes networks under fixed-point arithmetic. Given that modern networks use diverse activation functions (beyond ReLU and Step) and rely on floating-point arithmetic (as opposed to exact-real or fixed-point arithmetic), our results provide important new insights into the fundamental limits and capabilities of modern networks.

The paper does not provide an estimate for network width, which is crucial for practical applications.

As mentioned above, we focus on showing the existence of a floating-point network that can exactly represent a target floating-point function, as in existing universal approximation results (under real-valued parameters and operations). In this regard, the networks constructed in our paper are not for practical use. That said, we clarify the network size as follows. In the paper, we construct a $\sigma$-network of width $O(nd2^{(p+q)(d+1)})$, where $d$ is the dimension of an input and $n$ is a constant depending on the activation function $\sigma$ (e.g., $n \leq 10$ for ReLU, ELU, etc.). However, this size is to cover a general class of activations, and we believe a smaller width may suffice to obtain the same or similar results. Analyzing the minimum required width would be an important future direction.

On mixed-precision.

Our construction can be extended to mixed-precision setups. E.g., if the precision in the last layer can represent the target outputs and the domain is distinguishable under the precision of the first layer, then our construction can still represent any floating-point function even with low-precision intermediate layers, as long as they satisfy Condition 1 under the low precision.

In Theorem 3.4, the activation function and the domain of the floating-point function are coupled, making the conclusions somewhat ambiguous. What happens if the domain is the entire F?

The coupling between the activation function $\sigma$ and the domain is necessary, as shown in Lemma 3.2: if the domain (e.g., $\mathbb F$) is not $\sigma$-distinguishable with range $\mathbb F$ (Definition 3.1), then universal approximation using $\sigma$ networks is provably impossible. If the domain is the entire $\mathbb F$, Theorem 3.4 states that universal approximation is possible whenever $\mathbb F$ is $\sigma$-distinguishable with range $[-2^{e_{max}},2^{e_{max}}]$. This condition is satisfied, e.g., for $\sigma = \lceil\text{Sigmoid}\rfloor, \lceil\tanh\rfloor$ (Corollary 3.11).

On the depth of universal approximators.

Our current construction can be extended to five or more layers, but not to two or three layers. This is because our indicator function uses more than two layers and we use an additional layer to represent a linear combination of indicator functions. Investigating the minimum number of layers for universal approximation under floating-point arithmetic is an interesting research direction.

On multithreaded environment.

In our construction, we control round-off errors from floating-point operations by considering a single, fixed order of these operations (Sections 2.2–2.3). We expect that our results can be extended to multithread setups if the order of the operations in each layer is fixed (i.e., it does not depend on input values, memory size, etc.). However, if the order is not fixed, extending our results might be non-trivial due to the non-associativity of floating-point arithmetic.

The cos⁡ function is not periodic in computer arithmetic. Could this impact the conclusion of Lemma 3.3?

The $\lceil\cos\rfloor$ function in Lemma 3.3 is the rounded version of $\cos$. Thus, $\lceil\cos\rfloor$ is already aperiodic and the conclusion of Lemma 3.3 is valid for $\lceil\cos\rfloor$.

On “almost all” in the abstract.

Due to the 5000-character limit, we refer to our response to the same question from Reviewer EifD.

On related works.

Following the reviewer’s comment, we will include suggested related works on binary networks and sparse networks.

We would be happy to clarify any concerns or answer any questions, that may come up during the discussion period.