Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits

We thank the reviewer for deeming our paper to be novel, well-written, and for expecting it to have a high impact in the community. Below we reply to their points.

It is not clear to me what data set is used in the experiment of "Scaling PICs", and how many RVs are modelled.

In the experiment “Scaling PICs” (in the main text), whose results are shown in Fig 5, we used a PC modeling ImageNet 64x64. Note that it is not important which likelihood the PC assigns to different images, but only the time and space requirements for a forward/backward pass. Therefore, any batch of 64x64 RGB images would reproduce the same performance as reported in the plot. As we write in the caption, “The benchmark is conducted using a batch of 128 RGB images of size 64x64”. Finally, all RVs are categorical (256 categories for MNIST-like images, and 768 categories for RGB images), as described in lines 286-287.

About PICs with and without functional sharing

An extensive comparison between PICs with and without function sharing in terms of performance is not present in the paper, and below we explain why. However, luckily, as the reviewer pointed out, such comparison can be retrieved --- to a certain extent --- by taking results from Fig. 5 in [A], which corresponds to much smaller models than ours. We will account for it in the revised version of the paper.

As we extensively tested in our benchmarks (Fig. 5, Table D1, Table D2), PICs without functional sharing cannot scale to large model sizes, and therefore, simply because of that, PICs with functional sharing can deliver (better) results for the computational resources we have available.

This is in fact one of the main points of our paper: Scaling PICs with functional sharing, as we cannot without it. Therefore, given that PICs without functional sharing can unfortunately be run only for very small sizes (see Table D1 and D2), an extensive comparison in terms of performance w.r.t. PICs with functional sharing is not feasible on our GPUs.

We expect functional sharing to deliver better models as applying it drastically reduces the number of trainable parameters, and therefore possible overfitting. In Fig. 5 (top-right) we indeed showed that PICs without functional sharing hit 2B parameters, whereas with functional sharing only 6M.

[A] Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. AISTATS, 2024.

Could you elaborate why the trainable parameters in PIC (F, C) have reduced significantly compared with the PC (F, N)

Yes, the reason why there is a significant reduction in the number of trainable parameters between PIC (F, C) and PC (F, N) is because the former applies compositional sharing of MLPs, as explained in Section 3.3. However, once we materialize a QPC out of it, the number of its parameters match exactly the ones of PC (F, N), since these represent exactly the same architecture. With that in mind, we believe the comparison is fair as the PIC approach can be seen as a way of training PC (F, N).

For a detailed analysis, we refer the reviewer to our answer to reviewer WQnV about characterizing the size of PICs and PCs.

In the left most plots in Fig 5, I would infer that the single orange triangles in upper left are PIC(F,N) with QG-TK, but without a dotted line connected it is a bit unclear

Precisely, those isolated orange triangles refer to PIC (F, N) with QG-TK. The reason why they are isolated is because we could only run such configuration at K = 16, hence the lack of dotted lines. We refer the reviewer to Table D.2 for some tabular results of the plots, where this is evident. We will make sure to detail this better in the main text.

is the and before $||$ in line 156 a typo?

Yes, it should be and will be removed, thanks for spotting it.

We thank the reviewer for deeming our paper to be technically sound, clearly argumented, well-structured and easy-to-follow. Below we reply to their points.

it is currently not possible to sample from PICs, limiting their impact as a generative model.

It is true that since we use simple unconstrained MLPs to parameterize PICs --- which can be seen as energy-based models --- (differentiable) sampling from them is not possible. However, this is not a limitation of PICs per se, but a consequence of the way we choose to parameterize them. For instance, parameterizing using simple mixtures of Gaussians or even normalizing flows, would allow sampling from PICs.

Nevertheless, it is always possible to materialize PICs as QPCs and sample from them as usually done with PCs.

how significant is the tree-shape vs DAG-shape for empirical expressiveness of PICs? It appears that DAG-shaped PICs achieve slightly better bpds, but tree-shaped ones scale much better.

We remark that dropping 0.1 bpd on MNIST-like datasets would correspond to an increase of ~54 nats in log-likelihood, and perhaps this would be considered to be more significant. On RGB 64x64 datasets, the gap would even be much larger, i.e. ~851 nats.

It is true that there is a trade-off in terms of accuracy/efficiency. We just started exploring how to scale DAG RGs in a more effective way and believe our paper is a stepping stone in this direction. E.g., note that using Tucker layers yields the best performance for lower values of K, but we do not know how to scale it properly yet.

Section 3.2 describes a numerical quadrature rule as consisting of integration points and weights that minimize the integration error, and the authors later say that the same quadrature rule is used for each approximation. Does this raise any issues in approximation?

Using the same integration technique for each integral unit is not a limitation nor raises approximation issues when assuming a static quadrature regime. Assuming the same number of integration points, instead, allow us to tensorize the QPC in a regular way, thus promoting GPU parallelism via folding.

We conjecture that since we learn our functions through numerical approximation, these will stay nicely integrable w.r.t. to the points we used during training to fit them.

How crucial/limiting is the assumption that every integral unit integrates out all incoming latent variables

This is an interesting question! From a theoretical perspective, we would need to devise novel quadrature routines to materialize the QPC, as some latent variable would affect multiple integral units at once. In practice, it might be possible to “move” the non-integrated-out latent variables of the input function $g_i$ as to be part of variables $\mathbf{Z}_u$ of function $f_u$, for certain parameterizations of $g_i$ and $f_u$.

Nevertheless, we believe that from a learning perspective, PICs/QPCs have enough capacity to learn complex distributions with our assumption.

Do the QPCs approximating PICs still support tractable inference (e.g. marginal probabilities) similar to traditional PCs?

Yes, QPCs --- being composed of only sum, product and input units --- are proper PCs that inherit the structural properties of their corresponding PIC , e.g., they are also structured-decomposability if the underlying PIC is structured-decomposable. We will clarify this in the revised version of the manuscript.

Lines 133-134: do you also need Z_u3 != Z_u4 for CP-merge?

It does not need to be. If $Z_{u_3} \neq Z_{u_4}$, we would then merge u3 and u4 using Tucker-merge, whereas if $Z_{u_3} = Z_{u_4}$, we would then use again CP-merge. In this way, we allow for Tucker layers on top of CP-layers, and vice-versa.

Table 1: what does LVD-PG refer to?

It stands for Latent Variable Distillation - Progressive Growing, and it is a sophisticated learning scheme developed in [Liu, Xuejie, et al. "Understanding the distillation process from deep generative models to tractable probabilistic circuits." ICML 2023] that employs several heuristics to learn the parameters and structure of a PC on a lossy color space. We will specify this in the camera-ready version.

We thank the reviewer for deeming our paper to be a well-engineered solution delivering strong results. We are also happy that the reviewer enjoyed the reading. Below we reply to their questions.

The paper is arguably a little weak in terms of novelty

We respectfully disagree as it is non-trivial to recover a compact quadrature PC (QPC) from a PIC with a DAG structure. First we needed to change the original semantics of PICs/QPCs, which were defined only for tree region graphs, second, parameterizing the newly found tensorized QPCs needs some careful engineering otherwise the models would never be able to scale to the number of integration points we used (up to K=512).

What numerical quadrature rule was used?

We trained using the trapezoidal integration rule, we will document this in the revised version of the paper.

What is meant by differentiable [sampling] here, and is it possible to sample from the materialized PC if one is not interested in gradients (e.g. for visualization)?

When we say that differentiable sampling is not currently possible, it is just because the functions we use are parameterized as simple unconstrained MLPs, which can be seen as energy-based models, for which exact sampling is not possible. While using MCMC would be possible to sample from these neural networks, obtaining meaningful gradients through the samples would require additional approximations. Nevertheless, we can always materialize PICs as QPCs and sample from them as usually done with PCs.

To what extent do the authors consider their method a continuous latent variable model, as opposed to a means of effectively constructing PCs?

PICs are continuous latent variable models by definition. It is true, indeed, that the way we currently use PICs seems like a parameter learning procedure for tensorized PCs. However, we stress that at test-time we can actually materialize an arbitrarily sized PCs based on the number of integration points we use. We plan to explore (and exploit) this capability further in future work.

Did the authors investigate learning PICs according to a HCLT region graph?

Learning PICs using HCLT region graphs (RGs) has been already investigated in prior work [A]. In our work, we directly use RGs that, differently from HCLTs, are not learned from data but rather fixed for a given image size: QuadTree (QT) and QuadGraph (QG) RGs (see Appendix B for details). By comparing our results with the ones in [A], one can clearly see that QT and QG outperform HCLT, and highlight how learning an RG from images does not bring any advantage. QG outperforming QT brings evidence to support the claim that non-tree RGs can be more expressive.

[A] Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. AISTATS 2024

We thank the reviewer for deeming our paper to be a solid contribution, and a novel and efficient solution to an important problem. Below we clarify the confusion about notation, and respond to their points.

I'm confused by the definition of the integral unit from Line 58 to Line 62, especially by the inconsistency in the input of g_i between Line 61 and Line 62

$g_i(\mathbf{X}_i, \\{ \mathbf{Y}_i, \mathbf{Z}_u \\} )$ and $g_i(\mathbf{X}_i,\mathbf{Y}_i,\mathbf{Z}_u)$ are just equivalent notations, we will remove the brackets to avoid confusion. We used them trying to emphasize the set of $g_i$ LVs. Finally, note that, when writing an integral, the LVs that we integrate out are in lower-case (e.g. line 62)

the integral in Line 62 contains the function g_i while Figure 1(b) does not, which is also confusing. It might be helpful to put a numeric example there.

An integral unit $u$, by definition, receives input from a single unit, $i$. Assume that $u$ is associated with function $f_u(Z, Y) = 2Z - Y^2$, and that $i$ outputs the function $g_i(X, Y) = X^2 -3X + 4Y$. The integral unit would then compute the integral $\int_{-1}^{1} f_u(Z, y)g_i(X, y) dy$ (the integration domain is the support of $Y$, an assumption), therefore outputting the function $g_u(Z, X) = 2/3 (X - 3) X (6Z - 1)$, as we can compute the integral in closed-form in this example. (We will add this example to the camera-ready.) Therefore, in Figure 1, for instance, integral unit 5 (the red coloured one) computes $\int f_5(Z_2, z_4) f_4(X_4, z_4) dz_4$, i.e. $g_u = f_5$ and $g_i = f_4$.

It would be very helpful to provide a toy example to showcase the reduction of PICs to QPCs

This is a great suggestion and in the camera-ready version we will describe more in detail in the main text the example we provided appears in Figure 2. There, we illustrate the application of PIC2QPC (Algorithm 3) from the PIC in (b) to the QPC in (c), and then to its tensorized folded version in (d). Zooming in, we also illustrated the details of the materialization of a 3-variate function to a Tucker Layer in Figure 3. We will further expand this explanation and mention all the functions involved and comment on how Algorithm 3 operates step by step.

I'm curious to see some theoretical analysis on the circuit sizes characterized

We are not entirely sure how to understand the reviewer's request. We interpreted it as an analysis on the number of trainable parameters of PCs and PICs. We are happy to discuss more in case we are off-topic.

We address the answer by analyzing 3 types of folded layers (CP-layer, Tucker-layer, and Categorical-Layer), as our tensorized architectures are nothing more than a sequence of them. We use $F$ to denote the number of layers stacked together with folding, $K$ the number of integration points, $M$ the MLP size for PIC neural nets, and $L$ the number of MLP layers. We recall that the size of PIC neural nets is independent from $K$.

• A folded CP-layer is parameterized by a tensor of shape $F\times K\times K$ (where F is always even). Using PIC composite sharing, the size of the multi-headed MLP (assuming no bias term for simplicity) to parameterize such a layer is $LM^2 + FM$ (see Figure 4 for an illustration).
• A folded Tucker-layer is parameterized by a tensor of shape $F\times K\times K\times K$. Parameterizing such a layer with composite sharing, again requires an MLP of size $LM^2 + FM$.
• A folded Categorical-layer is parameterized by a tensor of shape $F\times K\times C$, where $C$ is the number of categories (256 of gray-scale images, and 768 for RGB ones). When using composite sharing, we can parameterize it with an MLP of size $LM^2 + FMC$. However, we found that PICs deliver better results when using full-sharing for the input layer, and this means using an MLP of size $LM^2 + MC$.

As an example, consider a folded CP-layer with $F=1000$, $K=512$, whose number of trainable parameters is $262,144,000$. Parameterizing it with an MLP with $L=2$ and $M=256$, delivers instead only $387,072$ trainable parameters, more than 99% less trainable parameters. We refer the reader to Appendix C1 for details about our MLPs, and to the plots on the right of Fig. 5 for the total trainable parameter counts of PC and PIC models. We will add this analysis in the revised paper.