We sincerely appreciate your careful and constructive review, which motivated us to perform additional experiments on ImageNet-1K embeddings and to analyze computational complexity explicitly. Your feedback has significantly strengthened our paper.

Response on Scalability and Computational Efficiency

We first address computational complexity to support the discussion on scalability.

Q2. Computational Cost

To analyze this clearly, we divide the computational requirements for any $c$-class classifier into two distinct parts:

1. Pre-softmax logit computation

2. Softmax computation

Let a Zero-Exclusive Network be a classifier that does not use the zero-vector class during training. Suppose there are $n$ training points, and let the computation needed to compute pre-softmax logits be $O(f(n))$—for instance, $f(n)$ might represent millions of computations. Computing the softmax function then takes $O(n \cdot \text{softmax}(c))$ computations, where $\text{softmax}(c)$ represents the complexity of softmax over $c$ classes.

Now consider modifying the Zero-Exclusive Network by adding an extra node at the output to classify the zero-vector class; we call this the Zero-Inclusive Network. In this scenario, we incorporate the zero-vector class in training. The addition of the new node introduces extra calculations dependent on the number of nodes in the preceding layer. Letting $L$ denote the number of nodes in the layer preceding this new node, the additional computation required is $O(nL^2)$.

The computational complexities during training and inference are summarized in the following table:

Typically, $O(f(n))$ dominates $O(nL^2)$ since $L$ corresponds only to the final internal layer. There is a trade-off between computational overhead and purity improvement. This trade-off depends significantly on the intended applications leveraging the full potential of the Zero-Vector framework.

We will include this complexity analysis in the additional page for the final submission.

Q1. Scalability

We acknowledge scalability as a critical concern. Our motivation for exploring high-dimensional embeddings comes from the success of embedding-based models like VQ-VAE, Stable Diffusion, and MAE, where internal dimensions (512–1024) capture complex distributions effectively.

To test this, we conducted experiments on embeddings (dim: 768) from a pretrained MAE on ImageNet-1K:

• Selected 10 random ImageNet classes and obtained embeddings using MAE
• Generated corresponding Zero-Vector Class data within this embedding space.
• Trained two classifiers:
  • Zero-Inclusive Network (trained with Zero-Vector Class data)
  • Zero-Exclusive Network (trained without Zero-Vector Class data)

Results:

• Purity Factor:
  • Zero-Exclusive Network purity fluctuated significantly (0.1–0.9), indicating substantial misclassification of empty regions in the feature space.
  • Zero-Inclusive Network purity remained consistently high and stable (0.7–1.0), indicating effective suppression of misclassification in empty regions.
• Accuracy:
  • Both networks reached similar test accuracy (~82%), showing no degradation from the Zero-Vector Class.

These experiments demonstrate the practical scalability of our framework to high-dimensional feature spaces, directly addressing the reviewer’s concern. Click here to view additional results on ImageNet-1K embeddings.

Similarly, in NLP Transformer models, next-token prediction is a classification over a fixed vocabulary, naturally allowing inclusion of the Zero-Vector Class as an extra token. While NLP experiments are left for future work, typical embedding sizes (~512) are smaller than those already tested (e.g., 784 for MNIST), supporting the method’s practical feasibility in NLP tasks.

Regarding References

This work explores a novel direction, and we found it challenging to identify direct precursors. Thank you for suggesting foundational works like word2vec and SVMs—we agree these are relevant in spirit and will incorporate such references in the final version. We welcome any further suggestions as well.

We believe these clarifications resolve the concerns raised. If you find these points convincing, we would appreciate a reconsideration of your Overall Recommendation. Thank you again for your valuable review and time.

We sincerely appreciate your careful and meticulous review, especially your identification of small errors with exact line numbers; this level of detail is invaluable for improving the manuscript's clarity and quality.

Clarification on Training Procedure for Clear Learning

Thank you for raising this important point. We acknowledge the training procedure was not fully detailed in the original submission and appreciate the opportunity to clarify.

Clear Learning follows a standard supervised classification setup using Cross-Entropy loss and SGD optimization. The only change is the inclusion of the Zero-Vector Class. This is done by uniformly sampling points from the input space, labeling them as the Zero-Vector Class, and adding them to the training data. The classifier’s output layer includes one extra node to represent this class.

All other components—architecture, loss function, optimizer, and hyperparameters—remain identical to baseline training. This makes the method easy to integrate into existing pipelines with minimal changes.

We used standard network architectures appropriate for each task:

• For MNIST and the 2D datasets (e.g., Figure~6), we used fully connected networks.
• For CIFAR-10, a conventional convolutional neural network.
• For the equation discovery task, a multidimensional Taylor-series-based classifier.

Given the range of tasks, we’ve included implementation code in the supplementary material and will release the full codebase upon acceptance. Training details will also be clarified in the final revision.

Assumptions about Zero-Vector Class (Mathematical Rigor & Clarity)

We appreciate your comment regarding the assumption that the Zero-Vector Class behaves like a uniform distribution and an additive identity. While our main goal was to convey the intuition leading to the framework, we agree that formalization is valuable. Your feedback helped us clarify this aspect more rigorously

As described in Section 2 (Characterization of Logits) of the main paper, if the data PDF is $f(x)$, a predefined threshold $\alpha_{f}$, can determine class membership. Thus tuple $(f(x), \alpha_{f})$ defines that class, say class$-f$. More generally, let $f_T(x)$ be the threshold of the PDF $f(x)$, and the tuple $(f(x), f_T(x))$ defines that class. Consider a black-box operator that takes a function as input and returns a threshold function suited for that PDF. For example, $(f(x) + k, f_T(x) + k)$ defines the same class-$f$ for any real-valued constant $k$. All such tuples preserve the decision boundary and thus represent the same underlying class.

Let 𝓜 be the set of monotonically non-decreasing functions. Then each tuple in the set $[f(x)]:=$ {$( M \circ f(x),\ M \circ f_T(x) ) \big| M \in 𝓜$ } also defines the same class-$f$. Here, $M \circ f(x)$ denotes $M(f(x))$, written this way to reduce bracket clutter.

Definition of set V: $V =${$[f(x)], [g(x)], [h(x)], ....$}

Addition:$\forall_{[f(x)], [g(x)] \in V}$ $[f(x)] + [g(x)] := [f(x) + g(x)]$

Scalar multiplication: $\forall_{\lambda \in \mathbb{R}}$ $\forall_{[f(x)] \in V}$ $\lambda [f(x)] :=[\lambda f(x)]$

The remaining vector space properties follow as in the main paper. Here, we focus on the additive identity (the zero-vector).

Let $[I(x)]$ is the identity of this vector space. Then, for any vector $[f(x)]\in V$

$[f(x)] + [I(x)] = [f(x)]$ ⇒ $[f(x) + I(x)] = [f(x)]$ ⇒ $[(f + I)(x)] = [f(x)]$ which implies:

{$(M ∘ (f + I)(x), M ∘ (f + I)_T(x)) | M ∈ 𝓜$} $=$ {$(M ∘ f(x), M ∘ f_T(x)) | M ∈ 𝓜$}

If $I ∈ 𝓜$, then the above holds trivially. Hence, the additive identity must be a monotonically non-decreasing function. Since every vector in $V$ corresponds to a class (i.e., a PDF), there must exist a PDF representing the zero-vector class that is monotonically non-decreasing. We now analyze its form:

Case 1: PDF of the Zero-Vector is Constant

A constant PDF corresponds to a uniform distribution. Hence, the zero-vector class corresponds to a uniform distribution over its support.

Case 2: PDF is Strictly Monotonically Increasing

Let $A$ be the volume spanned by a PDF that is strictly monotonically increasing and has zero probability outside the volume $A$. Since a PDF must integrate to one over its support, as $A \to \infty$, the PDF appears approximately constant within any finite, localized region. Thus, samples will appear uniform within any localized region.

From both Case 1 and Case 2, it follows that data sampled from the zero-vector appears uniform within any finite local region. Hence, the Zero-Vector Class effectively behaves as a uniform distribution.

We invite you to look at the computational cost and scalability discussed in response to reviewer gWUe. If our clarifications have addressed your concerns, we would be grateful if you would consider updating your overall recommendation. Thank you again for your time and thoughtful review.

Thank you for your thorough review and for positively highlighting the theoretical contributions of our work.

Q. Regarding Sampling from Low-Dimensional Manifolds

Your concern about sampling from low-dimensional manifolds is valid. Directly sampling from such manifolds is generally impractical. Crucially, however, our method explicitly does not require manifold sampling. Instead, we uniformly sample from the entire input space—a process that is both simple and ensures maximal entropy.

Why is this effective? Uniform sampling guarantees maximal entropy and thus explicitly represents regions of the input space associated with high uncertainty. These samples intentionally represent areas the classifier should not confidently assign to known classes. By introducing the Zero-Vector Class, we encourage the classifier to place decision boundaries precisely around true data manifolds, effectively learning the manifold structure indirectly. This is clearly demonstrated in Figure 6 (bottom row): without the Zero-Vector Class (bottom left), the classifier's boundaries incorrectly extend into empty regions. In contrast, with the Zero-Vector Class (bottom right), decision boundaries closely follow the true data distribution. This result validates our simple yet powerful sampling approach.

Regarding the Strength of Empirical Results

We respectfully disagree with the assertion that improvements from our method are marginal. The gains are substantial, measurable, and practically significant. Figure 6 visually demonstrates how the Zero-Vector Class dramatically improves decision boundary alignment with the true manifold. Moreover, quantitative evidence strongly supports our method:

• In Figure 18 (MNIST), the purity factor for classes 3 and 8 drops below 5% without the Zero-Vector Class, meaning the classifier confidently misclassifies 95% of the region. With the Zero-Vector Class, purity exceeds 95% implies misclassifies only 5% of the region.
• A similar pattern occurs in Figure 29 (CIFAR-10).

High purity explicitly indicates decision regions closely align with true data manifolds. Achieving this high purity directly enables a variety of applications (e.g., single-class learning, continual learning, equation discovery), which are otherwise infeasible with traditional classification methods.

Scalability to High-Dimensional Data

We appreciate your question about scalability, and explicitly evaluated it through additional experiments inspired by the practical success of embedding-based models (e.g., VQ-VAE, Stable Diffusion, and Masked Autoencoder [MAE]). Specifically, we:

• Selected embeddings (768-dimensional) from 10 random classes in ImageNet-1K using a pretrained MAE model.
• Generated corresponding Zero-Vector Class data uniformly in this embedding space.
• Trained two classifiers:
  • Zero-Inclusive Network (with Zero-Vector data)
  • Zero-Exclusive Network (without Zero-Vector data)

Results demonstrate clear scalability and effectiveness:

• Purity Factor:
  Zero-Exclusive Network purity fluctuates dramatically (0.1–0.9), indicating misclassification of empty space.
  Zero-Inclusive Network purity remains consistently high and stable (0.7–1.0).
• Accuracy: Both classifiers achieve comparable accuracy (~82%), confirming no accuracy penalty from the Zero-Vector Class.

These experiments clearly validate our method’s scalability to high-dimensional embedding spaces used by modern neural networks. Click here to view additional results on ImageNet-1K embeddings.

For NLP applications, Transformer models typically operate in embedding spaces with dimensions (~512) lower than those we have already successfully tested (e.g., MNIST: 784). Since next-token prediction in NLP is formulated as a vocabulary classification task, incorporating the Zero-Vector Class as an additional vocabulary token is straightforward and promising. Explicit NLP experiments will be pursued in future work.

We believe these clarifications resolve the concerns raised. If you find these points convincing, we would appreciate a reconsideration of your Overall Recommendation. Thank you again for your valuable review and time, and the insightful question.

We sincerely thank the Reviewer for their detailed and constructive review. The questions raised are insightful and have helped us further refine and improve our work.

Q1. Stability of Zero-Vector Class Training

We observed no instability during training with the Zero-Vector Class. The Zero-Vector Class uses identical network architecture and loss function as standard training. Empirically, we find training stability improved when incorporating the Zero-Vector Class:

• Purity Factor plots (e.g., "Figure 18 shows MNIST, Figure 29 shows CIFAR-10") are more stable during training when the Zero-Vector Class is included.
• Test Accuracy plot (e.g., "Figure 18 shows MNIST, Figure 29 shows CIFAR-10") is stable irrespective of zero-vector class used in training or not.
• No vanishing/exploding gradients and convergence issues as training metrics either improved or remained stable.

Q2. Differences from Energy-Based Models (EBMs)

We clearly distinguish our method from EBMs as follows:

• Representation: EBMs learn a single global energy function representing data likelihood.
  Our method explicitly learns separate class logits, each class individually via a standard classification network. This results in distinct, interpretable per-class distributions.

• Training Stability: EBMs typically require specialized training procedures (e.g., Langevin dynamics sampling), leading to potential instability or complex hyperparameter tuning.
  Our method uses standard supervised classification (cross-entropy loss) with the Zero-Vector Class, providing inherent training stability without extra sampling or specialized optimization.

• Generalization: EBMs are susceptible to mode collapse, potentially failing to capture the complete data distribution.
  Our method explicitly learns the underlying data distribution through class logits (Equation 27), converging to the true distribution as the number of data points increases. This ensures comprehensive generalization to the true data manifold, thus avoiding mode collapse.

Scalability and Computational Efficiency

Computational Efficiency

We analyze computational complexity in two parts:

1. Pre-softmax logits: complexity $O(f(n))$ for $n$ data points.
2. Softmax calculation: $O(n \cdot \text{softmax}(c))$ for $c$ classes.

Adding the Zero-Vector Class introduces one extra output node, adding overhead $O(nL^2)$, where $L$ is the preceding layer size.

Complexities summarized:

Typically, $O(f(n))$ dominates, making overhead manageable, especially given improvements in purity.

Scalability

Inspired by successful embedding-based models (VQ-VAE, Stable Diffusion, MAE; typical dimensions: 512–1024), we explicitly tested scalability on embeddings (dimension:768) from Masked Autoencoder (MAE, ImageNet-1K):

• Selected 10 random ImageNet classes, generated embeddings and corresponding Zero-Vector data.
• Trained Zero-Inclusive (with Zero-Vector data) and Zero-Exclusive classifiers.

Results confirm scalability:

• Purity Factors: 0.7–1.0 for Zero-Inclusive Network vs. 0.1–0.9 for Zero-Exclusive Network, demonstrating improved boundaries.
• Accuracy: Both networks achieved similar accuracy (~82%), confirming no accuracy degradation.

Click here to view additional results on ImageNet-1K embeddings.

In NLP Transformers, next-token prediction tasks classify over vocabulary, naturally permitting Zero-Vector Class integration. Typical NLP dimensions (~512) are lower than those already tested (e.g., MNIST:784), confirming practical feasibility. Explicit NLP experiments remain future work.

Response on Mathematical Rigor & Clarity:

Our goal was to clearly convey the core intuition behind our framework and support it empirically. We agree formalization is valuable and have provided rigorous mathematical explanations in response to Reviewer dsGt, which we encourage you to read for more details.

Regarding Essential References:

Given the novelty of this approach, identifying directly related prior work was challenging for us as well. We hope this contribution opens a new line of exploration.

We sincerely appreciate your detailed review and hope our responses clarified key concerns. Feedback from other reviewers was positive; if our clarifications resolved your concerns, we would appreciate your reconsideration of the Overall Recommendation. Thank you for your valuable evaluation and time.