We appreciate the reviewers’ thoughtful feedback and constructive suggestions. We addressed each point below.

[W1] The mathematical notations are understandable, but some parts, especially Equation 3, could be simplified to improve readability for a broader audience. Figure 1 could be clearer if it focused more narrowly on the core pipeline and provided additional details specific to the proposed framework. Figure 2 explains how the task vector is constructed but does not clarify whether the traversal produces variable-length vectors; this point would benefit from further elaboration. Also, Figure 3 feels disconnected from the main text, it would help to explain its context and relevance more explicitly.

We agree that improving the clarity of our notation and figures will make the core ideas of TVE more accessible. Below is how we will revise the manuscript to address each point:

• We will refactor Eq. 3 by introducing a compact notation for the set‐extraction step as follow:

  $$\mathbf{X}_v^{(t + 1)} = \\{ f\_{\mathcal{V}}(u) \mid u \in \mathcal{N}_k^{(t+1)} (v) \\}$$ $$y_v^{(t)} = l(\mathbf{X}_v^{(t + 1)})$$

  so that the reader sees at a glance “apply labeling function $l$ to the set of next-window k-hop neighbor set,” where we already make assumptions that these values are from a specific relation $R$. We also simplify the $\Delta t$ and represent the next-window variables via the upper-script $(t+1)$. We’ll move the full expansion to appendix remark for readers who want the full notations.

• Regarding Figure 1, we would like to provide a brief overview of different pre-training paradigms. We agree that Figure 1 could be clearer with more details describing the figure, so that readers can understand the general intuitions of our work. We will add more descriptions regarding inputs and outputs for each pre-training paradigm in Section 3.

• Regarding Figure 2, we will add a clarification on the variable-length vectors, as we answer below (cf. [Q1]).

• Figure 3 illustrates the Markov Chain that driving the label generation process. Specifically, for any set $\mathbf{Y}$ that is a generated by the parameter $\mathbf{\Theta}$, the downstream task is typically a labeling set function $l(\mathbf{Y})$. Our pre-training framework attempts to capture sufficient statistics $T(\mathbf{Y})$ of $\mathbf{Y}$ for $\mathbf{\Theta}$, which ultimately leads to task-aware latent representation. We agree that Figure 3 should be elaborated more, and we will add more details regarding Figure 3 in the camera-ready version.

[Q1] How do you ensure that the task vector maintains a consistent size across different traversals?

Our proposed task vector construction operates similarly to Breadth First Search, and tasks extracted from a schema path is concatenated to the final task vector (cf. Figure 2). While it true that different path leads to different length of task vector, the final objective is an estimation of the final task vector. We can enforce different traversal paths to have consistent length by leveraging data compression techniques like PCA or MCA, as we outlined in Section 4.2.

[Q2] The theoretical section discusses on SQL aggregation. In practice, the SQL queries might cover much more operators. How does this information loss affect learning effectiveness?

Thank you for raising this important point. We’d like to clarify complementary arguments showing why—and under what conditions—simple statistics can suffice, and how we mitigate the information loss when compared with sufficient statistics in practice.

Statistical sufficiency in common distributions.

In many distributions, a small collection of aggregators is provably sufficient. For example:

• The sum of samples is sufficient for exponential, Poisson, and Bernoulli distributions.
• The max is sufficient for a uniform distribution over a bounded interval.
• The mean is sufficient for a normal distribution with known variance.

Whenever the underlying dynamics match one of these canonical families, no information is lost by reducing the full set of observations to the corresponding statistic.

Enriching the task vector to minimize loss.

Real-world label‐derivation queries often deviate from textbook distributions. To guard against this, we augment our task vectors with as many simple statistics—include MEAN, MIN, MAX, SUM, COUNT, and STDDEV for numerical values, and MODE, COUNT, and COUNT DISTINCT for categorical values. This “overcomplete” summary ensures that, even if some statistics are redundant, we capture a broader range of moment and order information, thereby shrinking the worst‐case information gap.

Set‐theoretic recovery and downstream expressivity.

From a set‐theoretic standpoint, certain combinations of simple statistics can perfectly reconstruct small multisets. For instance, if exactly three purchases occur in the prediction window, then the triplet { MIN, MAX, SUM } uniquely determines the multiset. More generally, accurate pre‐training on these statistics helps the model learn an implicit approximation of the original value‐set, which in turn improves its ability to approximate any downstream labeling function—since most such functions operate on those same sets (cf. Eq. (3)).

Empirical validation in tabular domains.

Prior work on Deep Feature Synthesis (DFS) [1] demonstrates that stacking joins with simple aggregations often matches more expressive graph‐based architectures on many benchmarks [2, 3]. While DFS features incur theoretical information loss relative to a full relational graph encoding, their competitive accuracy shows that richly engineered statistics can suffice in practice. Our investigation is orthogonal: we systematically characterize all node‐level SQL‐generated labels as set functions and show how a task‐vector approach can act as surrogate representation for next-window sets. Additionally, our task vector includes similar aggregators to that of DFS.

Together, these points explain why simple statistics are not only theoretically grounded but also practically powerful—and how we effectively minimize any information loss.

[Q3] How complex are the expected SQL queries in practice? How many join hops and aggregations does the dataset typically involve?

Before deriving the computational complexity, assume the following:

• Number of rows per table is is $O(N)$.
• Number of timestamps is $T$.
• Branching factor of relational entity graph (the input graph for learning) is $b$.
• Number of aggregation functions is $r$.
• Branching factor of schema graph (the blueprint of the Relational Database) is $p$.
• Number of hops for generating the task vector is $k$.

We would like to compute the computational cost at each step for constructing the task vector. First, we compute the cost taken for a single path from the schema graph perspective (Step 1 - 3), and then sum up to get the final cost (Step 4). Remember that for every schema graph path, there are many paths fall under the same schema graph path since schema graph is just a blueprint of the input graph.

1. Joining with the time table, which is a necessary step regardless of pre-training paradigm, costs $O(NT)$.

2. Next, every join causes each row to fan out by a factor of $O(b)$, so the cost of joining up to $u$ successive times is:

   $$O(NTb + NTb^2 + ... + NTb^u) = O(NT\sum_{i=1}^u b^i).$$

   For $b > 1$, the geometric sum is scaled on the order of $b^u$:

   $$\sum_{i=1}^u b^i = \frac{b^{u+1} - b}{b-1} \approx \frac{b^{u+1}}{b-1} = \frac{b}{b-1}b^{u} = \Theta(b^u).$$

   Therefore, the total cost of for joining up to length $u$ is $O(NTb^u)$.

3. One aggregation of $r$ functions over the joined rows costs $O(rNTb^u)$, so every path (from the schema graph perspective) cost $O(NTB^u + rNTb^u) = O(rNTb^u)$ .

4. There are $O(p^u)$ paths of length $u$ from the schema graph perspective, summing the cost:

$$\sum_{u=1}^k \left[p^u \times O(rNTb^u)\right] = O\left(rNT\sum_{u=1}^k (pb)^u\right) = O(rNTp^kb^k)$$

Therefore, the cost blows up exponentially in $k$ through both $p^k$ and $b^k$, with linear dependence on $r$, $N$, and $T$. In practice, branching factor of the schema graph $p$ is typically small, and the schema graph is small, leading to small $k$. Furthermore, a path is only meaningful if it contains at least one fact table (the table containing timestamps), so that we can construct the predictive pre-training objective. Finally, this operation is only done once, and is not involved during pre-training optimization.

To demonstrate our point, we would like to report the time and peak memory usage for constructing task vector across datasets. As we can see from the below table, this operation is feasible even with large-scale databases, where rel-amazon and rel-hm contain approximately 24M and 33M rows, respectively. For our experiments, the maximum hop is 2, while the possible aggregation functions include MEAN, MIN, MAX, SUM, COUNT, and STDDEV for numerical values, and MODE, COUNT, and COUNT DISTINCT for categorical values. In industry scenario, such cost is expendable.

[1] Deep feature synthesis: Towards automating data science endeavors.

[2] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.

[3] Griffin: Towards a Graph-Centric Relational Database Foundation Model.

Dear Reviewer r84r,

We hope you’ve had a chance to read our rebuttal. We haven’t yet seen any follow-up questions or feedback from you in the discussion phase, so we wanted to check whether there are any points that remain unclear. We’d be grateful for any questions or thoughts you have—clarifying those will help us improve the paper.

Thank you again for your time,

The authors.

We are grateful to the reviewer for their feedbacks, and we have addressed each point below.

[Q1] The authors don't talk about the computational complexity...memory requirements…

Table 7 (Appendix B.6) shows each method’s pre-training time in seconds, demonstrating that TVE is relatively fast compared to others during pre-training. Next, we address computational complexity and memory requirements.

Before deriving the computational complexity, assume the following:

• Number of rows per table is $O(N)$.
• Number of timestamps is $T$.
• Branching factor of relational entity graph (the input graph for learning) is $b$.
• Number of aggregation functions is $r$.
• Branching factor of the schema graph (the blueprint of the Relational Database) is $p$.
• Number of hops for generating the task vector is $k$.

We derive the per-path cost (Steps 1–3) and then sum over all paths (Step 4). Remember that for every schema graph path, there are many paths that fall under the same schema graph path since a schema graph is just a blueprint of the input graph.

1. Joining with the time table, which is a necessary step regardless of pre-training paradigm, costs $O(NT)$.

2. Next, every join causes each row to fan out by a factor of $O(b)$, so the cost of joining up to $u$ successive times is:

   $$O(NTb + NTb^2 + ... + NTb^u) = O(NT\sum_{i=1}^u b^i) = O(NTb^u)$$

3. Aggregating $r$ functions adds $O(rNTb^u)$ per path.

4. There are $O(p^u)$ paths of length $u$ from the schema graph perspective, summing the cost:

$$\sum_{u=1}^k \left[p^u \times O(rNTb^u)\right] = O\left(rNT\sum_{u=1}^k (pb)^u\right) = O(rNTp^kb^k)$$

Therefore, the cost blows up exponentially in $k$ through both $p^k$ and $b^k$, with linear dependence on $r$, $N$, and $T$. Since $p$ and $k$ are small in real schemas, the exponential blow-up is limited. Furthermore, a path is only meaningful if it contains at least one fact table (the table containing timestamps), so that we can construct the predictive pre-training objective. Finally, this operation is only done once, and is not involved during pre-training optimization.

To demonstrate our point, we report the time and peak memory usage for constructing task vectors. As we can see from the below table, this operation is feasible even with large-scale databases, where rel-amazon and rel-hm contain approximately 24M and 33M rows, respectively.

[Q2] The authors should show results on some other benchmark…

First, we would like to clarify the core difference between RelBench [1] and 4DBInfer[2]. RelBench focuses on inference an SQL-generated label attached to rows in the dimension table (the table that is static and doesn’t contain timestamps), while 4DBInfer focuses on cell-level prediction.

We would like to show that our proposed framework is performant regardless of the evaluation method. We pre-trained TVE on seznam, and evaluated downstream performance on an entity attribute classification task. We reported 4 metrics— Accuracy, Macro F1, Micro F1, and Mean Reciprocal Rank (MRR)—on both validation and test splits. We include two separate tables, one for linear probing and one for full-finetuning.

seznam/charge (linear probing)

seznam/charge

We can observe that the performance of MAE and CTR is not consistent across datasets, while TVE consistantly performs well and exhibits competitive performance (matching MAE for linear probing and matching CTR for full-finetuning). This is because both MAE and CTR have limitations. For example, CTR doesn’t offer a separable representation after pre-training (low performance during linear probing), while MAE isn’t as performant as CTR during full fine-tuning. Moreover, these gains come despite a conceptual mismatch—TVE was optimized for next-state prediction on dimension tables, whereas the downstream tasks operate on fact-table rows. Regardless of evaluation setting, TVE consistently performs well across two different benchmarks RelBench and 4DBInfer.

[Q3] The authors must include more recent baselines…

Relational‐database–specific pre‐training remains an emerging area, and to our knowledge no paradigms beyond generic adaptations have been proposed. The only relevant study is [3]. Because of the substantial engineering required to adapt SSL methods for relational data contexts, we restrict our comparison to two representative branches of SSL, including generative and contrastive pre-training (cf. Appendix B.2 for detailed changes for adaptation). Through our study, we demonstrate that these SSL methods aren’t sufficient because they do not explicitly model next-window dynamics. Meanwhile, Task-Vector Estimation (TVE) explicitly models the schema-driven factors that govern downstream label generation. As the field matures, we look forward to new pre-training methods and will include them in our future evaluations.

[Q4] Authors should show the results on cross-domain transfer...

Unfortunately, cross-domain transfer is beyond the scope of our study; we focus on theoretical understanding of underlying factors driving downstream label generation process, and how to derive a task-aware pre-training objective. We adopted the baseline proposed in [1] for our study, which leads to different encoders for different relational databases.

Meanwhile, we believe that cross-domain transfer in the same database poses another interesting question. To better understand the separability of the pre-trained latent representation, we further conduct a linear probing experiment. TVE-1 (item) refers to the pre-trained model on the item's task vector, while TVE-1 (user) refers to the pre-trained model on the user's task vector.

Linear probing experiment:

Surprisingly, even when the task vector isn’t matched with downstream tasks’ entity type, it can still yield better performance than other generic SSL methods. For example, TVE-1 (user) is only worse than TVE-1 (item) on item-churn-least100-spending, and TVE-1 (item) is as strong as MAE-0.25 on user-churn-top100-spending.

Consequently, this experiment highlights the necessity to create an entity type-specific pre-training objective to maximize downstream performance gains.

[Q5] Expressions like "infinitely many possible downstream tasks" are imprecise…

We agree that “infinitely many possible downstream tasks” are imprecise. We will rephrase the term to be “diverse downstream tasks.”

[Q6] The paper is very difficult to read...

Below is how we will revise the manuscript to further enhance clarityr:

• We will refactor Eq. 3 by introducing a compact notation for the set‐extraction step as follow:

  $$\mathbf{X}_v^{(t + 1)} = \\{ f\_{\mathcal{V}}(u) \mid u \in \mathcal{N}_k^{(t+1)} (v) \\}$$ $$y_v^{(t)} = l(\mathbf{X}_v^{(t + 1)})$$

  where we already make assumptions that these values are from a specific relation $R$. We’ll move the full expansion to the appendix remark for readers who want the full notations.

• We agree that Figure 3 should be elaborated more, and we will add more details regarding Figure 3 in the camera-ready version.

We welcome any further clarity suggestions and thank the reviewers for their time.

[1] RelBench: A Benchmark for Deep Learning on Relational Databases.

[2] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.

[3] Flaky Performances when Pretraining on Relational Databases.

Dear Reviewer yYsy,

Thank you very much for acknowledging our rebuttal. We noticed that you didn’t have any follow-up questions or comments in the discussion phase. We would appreciate knowing whether our clarifications have sufficiently addressed your concerns. If anything remains unclear or you’d like further details, we’d be happy to elaborate.

Best regards,

The authors.

Dear Reviewer yYsy,

With the discussion period ending soon, I wanted to reach out and see if you have any outstanding questions or concerns about our paper and rebuttal. We’re happy to offer any clarification or information to ensure all your points are fully addressed.

Thank you again for your thoughtful review and time!

We would like to express our appreciation to the reviewers for the thorough evaluation and helpful recommendations. We would like to address the reviewer’s key concerns below.

[W1] While the paper claimed that TVE addresses task heterogeneity, the experiments mainly focus on performance on individual tasks rather than demonstrating transfer across multiple related tasks within the same database.

We would like to clarify that the goal of our work is to create a pre-training objective that can transfer across multiple downstream tasks like traditional pretraining in other fields such as language and image. Contrary to other fields, Relational Deep Learning offers a flexible framework where one can create a novel task that is based on temporality, structure, and SQL logics. Our goal is to systematically formulate that SQL-generated node-level labels are depending on sets of next-window values, and how to retain information of these sets during pre-training. In other words, we establish a universal mechanism for capturing supervisory signal of all downstream tasks in a single framework. As such, explicitly evaluating “bidirectional transfer” between two specific tasks lies beyond our core contribution, which is to prove and realize a unified pre-training paradigm that inherently supports transfer across the entire, potentially unbounded family of relational tasks.

[W2] It will be nice if the paper can provide more discussion about how to design better approximations of sufficient statistics.

Thank you for this perceptive suggestion. Designing richer approximations of sufficient statistics is indeed a vital and challenging direction, and we would like to engage in the discussion regarding possibilities of better representation for next-window sets.

Beyond fixed aggregators like task vectors, we can employ a small neural “set encoder” (e.g., DeepSets or transformer-based pooling) that learns to compress each next-window multiset into a vector. This approach can, in principle, capture all information in the set without hand-crafting statistics. However, training two networks—one to encode the set and one to predict the label—introduces instability. For instance, a GAN-style formulation (where the encoder acts as a generator and the predictor as a discriminator) risks mode collapse, mapping diverse inputs to the same latent code. Therefore, designing a sophisticated loss function to avoid mode collapse is a challenge for this strategy.

Our current task-vector objective optimizes only one encoder for input graph, which makes training stable and efficient. Despite being simpler, these vectors capture key moments of the distribution and are not susceptible to GAN-style collapse. They provide a reliable surrogate that practitioners can use immediately, while leaving more complex, learnable summaries to future work.

[W3] The paper does not analyze how well the chosen SQL aggregations approximate the ideal sufficient statistics of the underlying distribution.

Thank you very much for your thoughtful question. First, we would like to highlight that computing minimal sufficient representations has long been a grand challenge in both the Machine Learning and Information Theory communities. Because direct evaluation of mutual information is often intractable, Alemi et al. [1] instead optimize a tractable lower bound, while Belghazi et al. [2] turn to neural estimators to approximate mutual information directly.

Next, we would like to clarify complementary arguments showing why—and under what conditions—simple statistics can suffice, and how we mitigate the information loss when compared with sufficient statistics in practice.

Statistical sufficiency in common distributions.

In many distributions, a small collection of aggregators is provably sufficient. For example:

• The sum of samples is sufficient for exponential, Poisson, and Bernoulli distributions.
• The max is sufficient for a uniform distribution over a bounded interval.
• The mean is sufficient for a normal distribution with known variance.

Whenever the underlying dynamics match one of these canonical families, no information is lost by reducing the full set of observations to the corresponding statistic.

Enriching the task vector to minimize loss.

Real-world label‐derivation queries often deviate from textbook distributions. To guard against this, we augment our task vectors with as many simple statistics—include MEAN, MIN, MAX, SUM, COUNT, and STDDEV for numerical values, and MODE, COUNT, and COUNT DISTINCT for categorical values. This “overcomplete” summary ensures that, even if some statistics are redundant, we capture a broader range of moment and order information, thereby shrinking the worst‐case information gap.

Set‐theoretic recovery and downstream expressivity.

From a set‐theoretic standpoint, certain combinations of simple statistics can perfectly reconstruct small multisets. For instance, if exactly three purchases occur in the prediction window, then the triplet { MIN, MAX, SUM } uniquely determines the multiset. More generally, accurate pre‐training on these statistics helps the model learn an implicit approximation of the original value‐set, which in turn improves its ability to approximate any downstream labeling function—since most such functions operate on those same sets (cf. Eq. (3)).

Empirical validation in tabular domains.

Prior work on Deep Feature Synthesis (DFS) [3] demonstrates that stacking joins with simple aggregations often matches more expressive graph‐based architectures on many benchmarks [4, 5]. While DFS features incur theoretical information loss relative to a full relational graph encoding, their competitive accuracy shows that richly engineered statistics can suffice in practice. Our investigation is orthogonal: we systematically characterize all node‐level SQL‐generated labels as set functions and show how a task‐vector approach can act as surrogate representation for next-window sets. Additionally, our task vector includes similar aggregators to that of DFS.

Together, these points explain why simple statistics are not only theoretically grounded but also practically powerful—and how we effectively minimize any information loss.

[Q3] How sensitive is TVE to the quality of the schema graph representation? Would errors in schema understanding significantly impact performance?

Thank you for this excellent question. To quantify TVE’s sensitivity to schema‐graph quality, we ran controlled “noise” experiments on the RelBench rel-amazon database:

1. We randomly dropped 20% and 40% of the rows in the review table to corrupt the relational entity graph. The corrupted graph here is considered clean signals, and tasks are depending on this newly created graph.
2. We treat the unaltered database as noisy database; thus, the downstream tasks defined in 1. no longer fully dependent on the original schema graph. Therefore, a robust pre-training paradigm overfitting to the noisy relational entity graph won’t generalize well to the tasks in 1. It simulates the case when tasks are not based on the schema graph due to errors in schema understanding.
3. We compared three pre-training paradigms—MAE, CTR, and our TVE—on two tasks user-churn-top100-spending and user-churn. We then (a) pre-trained our models on the noisy graph (including task vector constructed from this noisy graph), and (b) fine-tuned (and evaluated) on the same noisy graph using labels generated from the clean database.

rel-amazon/user-churn-top100-spending (linear probing)

rel-amazon/user-churn-top100-spending

rel-amazon/user-churn (linear probing)

rel-amazon/user-churn

Even when the downstream task loosely depends on schema graph understanding, and when TVE is pre-trained on a noisy relational database, TVE’s task‐aware pre-training consistently outperforms standard SSL objectives. This demonstrates that our framework is robust to errors in schema understanding and retains its effectiveness across a range of noise levels.

[1] Deep Variational Information Bottleneck.

[2] Mutual Information Neural Estimation.

[3] Deep feature synthesis: Towards automating data science endeavors.

[4] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.

[5] Griffin: Towards a Graph-Centric Relational Database Foundation Model.

Dear Reviewer 5cD2,

Thank you very much for acknowledging our rebuttal. We noticed that you didn’t have any follow-up questions or comments in the discussion phase. We would appreciate knowing whether our clarifications have sufficiently addressed your concerns. If anything remains unclear or you’d like further details, we’d be happy to elaborate.

Best regards,

The authors.

Dear Reviewer 5cD2,

As the discussion period wraps up, I wanted to check in and see if you have any remaining questions or concerns about our paper and rebuttal. We’d be happy to clarify or expand on any points to ensure everything is addressed.

Thank you again for your time and feedback!