Dolphin: A Programmable Framework for Scalable Neurosymbolic Learning

We thank the reviewer for their insightful feedback and the extensive literature they brought to our attention. We will include a deeper discussion in the related work section on the systems from the literature and clarify other points within the revised manuscript.

Novelty

Tensor Operations for Neurosymbolic Learning.

We thank the reviewer for pointing out the literature on parallelized neurosymbolic learning. We agree that the concept of using tensor operations for neurosymbolic learning is not new. However, systems such as LYRICS, Logic Tensor Networks (LTNs), and Tensorlog all have limited expressivity, which is one of the obstacles Dolphin aims to overcome. Specifically, they restrict the symbolic programs to first order logic and require the user to specify low-level information such as how variables are grounded and what their domains are. They also restrict the symbols to be in the form of tensors and the user-defined functions to consist of TensorFlow operations. These restrictions allow such systems to use TensorFlow to compile these programs into highly efficient computational graphs, but at the cost of expressivity. These frameworks also exclusively support simpler provenances and t-norms which are not sufficient for complex neurosymbolic programs. We describe these systems briefly in the related work section of the revised paper.

On the other hand, Dolphin allows the user to track tags for specific symbols which can be arbitrary Pythonic objects. Dolphin programs further allow the user to manipulate Distributions over such symbols using arbitrarily complex code which may not necessarily translate to a computational graph. As such, there is a fine balance between the probabilistic computations, that happen over a GPU, and the symbolic computations, that take place on a CPU, all while maintaining a mapping between the two. This requirement sets a unique challenge addressed by Dolphin that we believe sets it apart from systems that use tensor operations for neurosymbolic learning. This fundamental design choice is also what allows Dolphin to be more expressive and flexible than existing systems. We also design Dolphin to be modular so that users can easily extend it to support new provenances and t-norms. As such, the t-norms used in LYRICS and LTN can be trivially added in a vectorized manner to Dolphin. We have added this discussion in the revised manuscript, along with a comparison of the expressivity of Dolphin with LTN for MNIST Sum-2, and demonstrate why HWF-N as written in Dolphin is not feasible in LTN. Please refer to Appendix E.

Optimizing Probabilistic Computations via Tensors.

Other works pointed by the reviewer, such as Dang et al. and Darwiche, focus solely on probabilistic computations rather than neurosymbolic frameworks. For instance, Juice by Dang et al. is a Julia package for logic and probabilistic circuits, which is not designed to be integrated with deep learning frameworks. On the other hand, Darwiche's work focuses on variable elimination with applications to optimize tensor-based computation. It will be interesting to see how Dolphin can be integrated with such systems to further improve the scalability and efficiency of neurosymbolic learning, and we will include a discussion on this in the revised manuscript. However, we still believe that Dolphin's novelty lies in its design that allows for the seamless integration of expressive neurosymbolic programs within deep learning frameworks, which is not addressed by the existing systems. We add this discussion in Appendix E.

About the likelihoods

We remove the “relative” term from the current revision for describing the likelihoods, since we do not normalize the likelihoods within Distributions. Typically, any normalization would occur in a neural network via functions like the sigmoid or softmax functions. But we do not enforce any normalization explicitly within the Dolphin program or the Distributions themselves. We agree that calling them Distributions may be confusing and incorrectly imply that they are specifically probability distributions. We will change the name to something less confusing if the reviewer suggests it.

About the Scallop results

We have not seen any behavior to indicate that there is a bug in Scallop’s backend. It is more likely the case that PyTorch performs optimizations that result in a computation graph that is easier to converge over.

About the IndeCateR+ results

We did notify the authors of the ISED paper, and they have updated their numbers as well.

Scaling in the Presence of Sequential Symbol Computations

Given that we do not compile symbol computations into PyTorch computation graphs to satisfy our first design principle, satisfying the third design principle (scalability) becomes a challenge. We address this by condensing symbols in the Distribution class into a single collection stored in CPU RAM while maintaining tags as a GPU tensor ($b \times N \times T$, where $b$ is the batch size, $N$ is the number of symbols, and $T$ is the shape of the tag) as described in our response to Reviewer mznt. This results in there being one set of CPU-based computations for the entire batch of samples rather than one set of computations for each sample within the Distribution, which is typical of other neurosymbolic frameworks. This allows Dolphin to maintain the benefits of parallelism even while the user-defined functions are executed sequentially.

We demonstrate this by sharing the breakdown of the time taken for symbol computations and tag computations within the forward pass of the Dolphin program for the HWF task that we showed in the response to Reviewer mznt. The first row shows the time taken during the forward pass when the Dolphin program is run sequentially on the CPU with no parallelism. The second row shows the time taken when tag computations are parallelized on the GPU over batches of 64 samples each. The times annotated with C and G indicate time spent on the CPU and GPU, respectively:

This also means that due to Dolphin’s design, increases in batch size result in fewer total CPU operations over the entire training epoch, since the set of CPU operations is shared for the entire batch while parallelizing more tag computations over the entire batch. We observe similar behavior for MNIST Prod-N (Table 4 in Appendix F), where as we increase the batch size, the training time reduces for large values of N.

Addressing Tunability

These design choices also serve to satisfy the last design principle of tunability: since Dolphin decouples the symbol computations from the probability computations, we are able to treat provenances as another hyperparameter in the deep learning pipeline, and even allow developers to add more provenances without changing the Dolphin framework itself.

HWF-N in LTN

We do not claim that one cannot write HWF using LTN. What we intend to say is that to write the same HWF program as Dolphin in LTN, LTN needs to support strings as constants and allow programmers to supply arbitrary Python functions like eval. As discussed earlier, LTN does not allow either. So in order to solve HWF, one must write a much less intuitive and more complex program.

About the Choice of Primitives

We thank the reviewer for their feedback on Appendix H, and we will add more details in a revised manuscript. Is there something specific the reviewer would like to see in those details?

That said, the fact that one can supplement these primitives with user-defined functions means that these primitives can be easily used to write programs for other tasks.

About Expressivity in terms of Scallop

We say that we are as expressive as Scallop because Scallop offers more features than Datalog, such as foreign functions and algebraic data types.

About ApplyIf

Adding optimizations to the computation graph generated by Dolphin is indeed an interesting area of research. We leave this to future work since it is orthogonal to the design of a general-purpose neurosymbolic framework.

About DTKP-AM

We do specify in the revised paper that DTKP-AM is an approximation of DTKP-WMC. We also do not claim to provide any guarantees, and have clarified our appendix to say that DTKP-AM only upper bounds DTKP-WMC. We test our both DAMP and DTKP-AM on the following example that does not involve mutually exclusive proofs:

a = Distribution(self.mnist_net(a_imgs), range(10))
res = a + a + a + a

Using DTKP-AM results in a 98.86% accuracy, while using DAMP results in a 90.51% accuracy. While this is a simple example, it does show that DTKP-AM does provide benefits when the proofs in the symbolic computation are not mutually exclusive.

We thank the reviewer for taking the time to engage with us and helping us improve the presentation and clarify our contributions.

About the Relationship of Dolphin with LTN

We apologize for the confusion as we try to understand the relationship of Dolphin to LTN as it pertains to the concerns raised by the reviewer. We clarify our contributions with respect to the four design principles outlined in Section 3.1 using the simplest example, MNIST.

The Dolphin Program for MNIST Sum-2

We show the Dolphin program for MNIST Sum-2 here:

d1 = Distribution(model(img[0]), range(10))
d2 = Distribution(model(img[1]), range(10))

result_logits = GetProbs(Apply(d1, d2,  lambda x, y: x + y))

There exist two kinds of computations in Dolphin: those occurring over symbols and those occurring over their corresponding probabilities. Symbols can be any objects (e.g. here the digits 0, 1, …, 9), and functions over them can be arbitrary operations (here the addition function).

As we state in our first design principle, Dolphin allows for flexible programmability. To enable this, the symbolic computations (e.g. $f_\text{add}(1, 2)$) are run as sequential Python code on the CPU. This enables symbolic computations to be arbitrarily complex functions $f$ expressed in a high-level language like Python.

To preserve this flexible programmability, Dolphin does not compile symbols and functions over them into PyTorch computation graphs. Doing so would require restricting the symbols to be tensors, and restricting the functions to be a chain of PyTorch operations over those tensors.

On the other hand, Dolphin does compile the probability computations (e.g. d1(1) $\otimes$ d2(2)) over those symbols into PyTorch computation graphs that can be heavily parallelized, since the probabilities themselves are tensors on the GPU (assuming the model is run on the GPU), thus satisfying our second design principle of end-to-end differentiability.

The LTN Program for MNIST Sum-2

In contrast to Dolphin, LTN compiles both the symbol computations and the probability computations into TensorFlow computation graphs. The LTN program for MNIST Sum-2 is as follows:

### Predicates
Digit = ltn.Predicate.FromLogits(model, activation_function="softmax")
### Variables
d1 = ltn.Variable("digits1", range(10))
d2 = ltn.Variable("digits2", range(10))
### Operators
Not = ltn.Wrapper_Connective(ltn.fuzzy_ops.Not_Std())
And = ltn.Wrapper_Connective(ltn.fuzzy_ops.And_Prod())
Or = ltn.Wrapper_Connective(ltn.fuzzy_ops.Or_ProbSum())
Implies = ltn.Wrapper_Connective(ltn.fuzzy_ops.Implies_Reichenbach())
Forall = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMeanError(),semantics="forall")
Exists = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMean(),semantics="exists")


# mask
add = ltn.Function.Lambda(lambda inputs: inputs[0]+inputs[1])
equals = ltn.Predicate.Lambda(lambda inputs: inputs[0] == inputs[1])

### Axioms
@tf.function
def axioms(images_x, images_y, labels_z, p_schedule=tf.constant(2.)):
	images_x = ltn.Variable("x", images_x)
	images_y = ltn.Variable("y", images_y)
	labels_z = ltn.Variable("z", labels_z)
	axiom = Forall(
        	ltn.diag(images_x,images_y,labels_z),
        	Exists(
            	(d1,d2),
            	And(Digit([images_x,d1]),Digit([images_y,d2])),
            	mask=equals([add([d1,d2]), labels_z]),
            	p=p_schedule
        	),
        	p=2
    	)
	result_logits = axiom.tensor
	return result_logits

This does not satisfy our first design principle due to the reasons mentioned above: constants in LTN programs have to be grounded as tensors rather than remaining arbitrary Python objects, and the functions have to be compilable into a TensorFlow computation subgraph. Note that user-defined functions need to be supplied using ltn.Function.Lambda or ltn.Predicate.Lambda, which can only accept expressions over tensors rather than Python functions over arbitrary objects.

This is the fundamental difference between Dolphin and systems like LTN and Scallop. On one hand, similar to LTN, Dolphin uses tensor computations and GPU support to enhance scalability compared to Scallop. On the other hand, as the reviewer also noted, Dolphin is more intuitive, allowing programmers to write symbolic computations over dynamic data structures in a high-level language like Python.

> It is true that the add-mult step in DTKP-AM is less precise than using WMC. However, this does not destroy the top-k semantics.(…)

I get why the authors go for this approximation (it’s much easier than parallizing the actual probabilistic inference), but the paper should be clear about this. E.g. Appendix C states “We note that this approximation upper bounds the result from “full” WMC”, but really you have a upper bound on the top-k which is itself a lower bound on the full WMC. (i.e. there rema no guarantee at all). “We even hypothesize that in most cases, the add-mult approximation does not meaningfully affect the final result compared to full DTKP”; is this a different way to say Dolphin only considers tasks with mutually exclusive proofs (as e.g. Winters et al. targets)? Otherwise, there is considerable evidence in the literature that differentiating fuzzy semantics can be problematic (see e.g. van Krieken et al.).

Winters, T., Marra, G., Manhaeve, R., & De Raedt, L. (2022). Deepstochlog: Neural stochastic logic programming. In Proceedings of the AAAI Conference on Artificial Intelligence.

van Krieken, E., Acar, E., & van Harmelen, F. (2022). Analyzing differentiable fuzzy logic operators. Artificial Intelligence.

> We originally used the IndeCateR+ implementation provided by the ISED authors in their artifact, which was a CPU-centric implementation that was severely undersampled. We have rerun the MNIST and HWF experiments using the correct implementation and have updated the result tables as well as the text in the revised manuscript.

We thank the authors for clarifying this. I was somewhat surprised to hear that the ISED implementation of indecater would perform so much worse, as the reported accuracies in the ISED paper are much higher than what the Dolphin paper reported. If there was an error in the ISED paper, perhaps this should be communicated to the authors?

> The tags represent the likelihoods for each symbol in a Distribution relative to other symbols in the same Distribution.

But if this is a relative likelihood (as stated in the paper), what is the base likelihood you normalize with? Relative likelihood is a likelihood divided by another likelihood 1.

> Yes, Distributions are not probability distributions, but simply a mapping from a set of symbols to their likelihoods

So why call it a distribution if it’s not a distribution? And more importantly, w.r.t. what distribution is this likelihood of a symbol?

More generally, I don’t understand why the paper and rebuttal talks about probabilities and likelihoods at several points while the authors at the same time also say that Dolphin does not have probabilistic semantics.

> We therefore hypothesize that Dolphin converges to a higher accuracy because it uses PyTorch for differentiating symbolic programs, while Scallop uses its own auto-differentiation framework.

Well the autodiff framework shouldn’t make any difference, unless the authors are implying that Scallop has errors in their autodiff implementation? This might be worth mentioning in the paper.

I want to thank the authors for their extensive clarifications. However, I do not feel that my concerns about the paper are adequately addressed. I hence maintain my score.

We thank the reviewer for their feedback and will incorporate it to make the paper more readable and easily understandable. We address the reviewer’s concerns below:

About the challenges outlined in the paper

Using a separate CPU-based backend is not merely an implementation choice but a fundamental design limitation in most neurosymbolic frameworks. As mentioned in the introduction, these frameworks must perform symbolic computations while tracking tags and probabilities across input, intermediate, and output symbols, which inherently involve complex, variable-sized data structures and operations that are difficult to parallelize.

This limitation forces existing frameworks to rely on CPU backends. Since deep learning models are typically implemented in Python, frameworks face a trade-off: either implement symbolic reasoning directly in Python, which is slow for CPU-based operations or use a separate backend, typically written in some compiled language. While the latter is faster, it requires transferring data structures, operations, logits, and gradients between the neural pipeline and the external backend, introducing inter-process latency. We clarify the reason of interprocess data transfers in the introduction.

About the core principles

Dolphin tries to address the following core principles as follows:

• Flexible programmability: The Distribution abstraction, along with the associated primitives, allow for expressing complex neurosymbolic programs over arbitrary Python objects through user-defined Python functions.
• End-to-end differentiability on GPUs: The Distribution abstraction maintains a mapping from symbols to tags stored as PyTorch tensors on GPUs sourced directly from neural network models. Furthermore, the provenances governing the tag operations are defined in a differentiable manner, allowing Dolphin to harness PyTorch’s GPU support and auto-differentiation mechanisms.
• Scalability: Operations over Distributions defined through its primitives can be batched easily, which are processed in a vectorized manner, allowing Dolphin programs to scale with larger and more complex datasets.
• Tunability: The modular design of Distributions, where the primitives are defined independent of the provenances, allows users to rapidly plug in and test out different provenances to select the one best suited for their task.

Together, these core principles directly address the challenges of both problem complexity as well as data complexity while inhibiting the need for a separate CPU-based backend. Principles 1 and 4 address the issue of program complexity, while principles 2 and 3 focus on scaling with data complexity. We describe how these principles map to the challenges in the revised manuscript in Section 3.1.

Other Questions

Why does Dolphin have a better accuracy?

We attribute the difference in accuracies between Dolphin and ISED / IndeCateR+ to the underlying design of each baseline. IndeCateR+ and ISED are sampling-based gradient approximation methods, which are inherently stochastic and may not converge to the optimal solution. As for Scallop, this only happens in Mugen, where we write the same program as Scallop’s in Dolphin and use the same base neural network. The only difference here is in the backend neurosymbolic engine. We therefore hypothesize that Dolphin converges to a higher accuracy because it uses PyTorch for differentiating symbolic programs, thus benefitting from Python’s optimizations over the computational graph, while Scallop uses its own auto-differentiation framework.

How should we select the most suitable provenance in practice?

The choice of provenance depends on many factors, including the complexity of the program, the independence of variables within the program, and the desired trade-off between accuracy and training time. Typically, one would try each provenance and use the one yielding the best accuracy. If both provenances perform similar, it is more practical to choose DAMP over DTKP due to its efficiency.

About the breakdown of training times.

Unfortunately, finding the time taken for the inter-process data transfer requires profiling systems like Scallop in detail, which is not readily available.

d1 = Distribution(model(img[0]), range(10))
d2 = Distribution(model(img[1]), range(10))

result_logits = GetProbs(Apply(d1, d2,  lambda x, y: x + y))

### Predicates
Digit = ltn.Predicate.FromLogits(model, activation_function="softmax")
### Variables
d1 = ltn.Variable("digits1", range(10))
d2 = ltn.Variable("digits2", range(10))
### Operators
Not = ltn.Wrapper_Connective(ltn.fuzzy_ops.Not_Std())
And = ltn.Wrapper_Connective(ltn.fuzzy_ops.And_Prod())
Or = ltn.Wrapper_Connective(ltn.fuzzy_ops.Or_ProbSum())
Implies = ltn.Wrapper_Connective(ltn.fuzzy_ops.Implies_Reichenbach())
Forall = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMeanError(),semantics="forall")
Exists = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMean(),semantics="exists")


# mask
add = ltn.Function.Lambda(lambda inputs: inputs[0]+inputs[1])
equals = ltn.Predicate.Lambda(lambda inputs: inputs[0] == inputs[1])

### Axioms
@tf.function
def axioms(images_x, images_y, labels_z, p_schedule=tf.constant(2.)):
	images_x = ltn.Variable("x", images_x)
	images_y = ltn.Variable("y", images_y)
	labels_z = ltn.Variable("z", labels_z)
	axiom = Forall(
        	ltn.diag(images_x,images_y,labels_z),
        	Exists(
            	(d1,d2),
            	And(Digit([images_x,d1]),Digit([images_y,d2])),
            	mask=equals([add([d1,d2]), labels_z]),
            	p=p_schedule
        	),
        	p=2
    	)
	result_logits = axiom.tensor
	return result_logits

Scaling in the Presence of Sequential Symbol Computations

Given that we do not compile symbol computations into PyTorch computation graphs to satisfy our first design principle, satisfying the third design principle (scalability) becomes a challenge. We address this by condensing symbols in the Distribution class into a single collection stored in CPU RAM while maintaining tags as a GPU tensor ($b \times N \times T$, where $b$ is the batch size, $N$ is the number of symbols, and $T$ is the shape of the tag) as described in our response to Reviewer mznt. This results in there being one set of CPU-based computations for the entire batch of samples rather than one set of computations for each sample within the Distribution, which is typical of other neurosymbolic frameworks. This allows Dolphin to maintain the benefits of parallelism even while the user-defined functions are executed sequentially.

We demonstrate this by sharing the breakdown of the time taken for symbol computations and tag computations within the forward pass of the Dolphin program for the HWF task that we showed in the response to Reviewer mznt. The first row shows the time taken during the forward pass when the Dolphin program is run sequentially on the CPU with no parallelism. The second row shows the time taken when tag computations are parallelized on the GPU over batches of 64 samples each. The times annotated with C and G indicate time spent on the CPU and GPU, respectively:

This also means that due to Dolphin’s design, increases in batch size result in fewer total CPU operations over the entire training epoch, since the set of CPU operations is shared for the entire batch while parallelizing more tag computations over the entire batch. We observe similar behavior for MNIST Prod-N (Table 4 in Appendix F), where as we increase the batch size, the training time reduces for large values of N.

Addressing Tunability

These design choices also serve to satisfy the last design principle of tunability: since Dolphin decouples the symbol computations from the probability computations, we are able to treat provenances as another hyperparameter in the deep learning pipeline, and even allow developers to add more provenances without changing the Dolphin framework itself.

About the potential of combinatorial explosions in computations

Handling Combinatorial Explosions.

We thank the reviewer for highlighting the issue of combinatorial explosion. This is indeed a fundamental challenge in neurosymbolic programs as a whole. Dolphin mitigates this by leveraging the Distribution class, which condenses symbols into a single collection stored in CPU RAM while maintaining tags as a GPU tensor (b x N x T, where b is the batch size, N is the number of symbols, and T is the shape of the tag). As shown in Figure 2(b), this approach reduces symbolic overhead by avoiding redundant evaluations for each sample in a batch, unlike frameworks like Scallop, where each sample is independently evaluated. While tag evaluations still involve all combinations across all samples in a batch, they are computed in a vectorized manner on the GPU. We include this explanation in Appendix F.

Results for MNIST Product-N.

We conducted a preliminary experiment for the MNIST Product-N benchmark suggested by the reviewer. We summarize the results below. For each N, we report the time taken per epoch in seconds averaged over 5 epochs as well as the accuracy achieved:

We get further scalability improvements by increasing the batch size. Increasing the batch size from 64 to 128 yields these numbers:

We see the effect of combinatorial explosion as the time taken per epoch increases with N. However, the explosion does not render the computation infeasible, and Dolphin is still able to achieve high accuracy. The runtime also scales within reason, and increasing the batch size reduces the runtime due to batched computations within Dolphin. We include these results in Appendix F.

How does Dolphin deal with control flow?

Control Flow in Dolphin.

In Dolphin, control flow largely exists within the lambda functions supplied to the Apply, ApplyIf, and Filter operations, which can be arbitrary Python functions over the symbols in the Distributions. As discussed in Section 3.2.2, these functions can include complex operations like if-then-else branches, loops, and even recursion. The nature of these functions means that they cannot be parallelized over the GPU. Instead, they are executed sequentially on the CPU, while the associated tags are computed parallely on the GPU. We optimize the design of the Distribution class so that there is one set of CPU-based computations for the entire batch of samples rather than one set of computations for each sample, which is typical of other neurosymbolic frameworks. This allows Dolphin to maintain the benefits of parallelism even while the user-defined functions are executed sequentially. We include this discussion in Appendix D.

Control Flow for HWF.

We demonstrate how Dolphin handles control flow by showing the time taken for the HWF task split by the time spent on the CPU and GPU. The code for this task is shown in Appendix G, and involves complex control flows within its user-defined functions (UDFs) like Python’s eval operation. The first row shows the time taken during the forward pass when the Dolphin program is run sequentially on the CPU with no parallelism. The second row shows the time taken when tag computations are parallelized on the GPU over batches of 64 samples each. The times annotated with C and G indicate time spent on the CPU and GPU, respectively:

Observe that the time, both for UDF computation and for Tag computation, decreases as we move from sequential CPU evaluation to the batched evaluation. Due to Dolphin’s design, increases in batch size result in fewer total CPU operations over the entire training epoch, since the set of CPU operations is shared for the entire batch, while parallelizing more tag computations over the entire batch. We include these results in Appendix D as well.

Recursion.

In order to write recursive computations in Dolphin, one has two choices: either supply a recursive user-defined function to the Dolphin primitives, or write a more fine-grained program in Python that uses Dolphin primitives in the base case as well as the recursive case, set to terminate once a condition is met. Here, the diverging control flows can be merged using the Union primitive. We discuss recursion and control flow further in Appendix D and show an example on writing recursive programs using Dolphin.

About the HWF task

We thank the reviewer for pointing out that Dolphin currently only supports deterministic symbolic processes. Before we address this, we would like to explain how the HWF example as suggested by the reviewer can be implemented in Dolphin. In the HWF task presented in the paper, the neural model does predict both numbers and operators by classifying each image into 14 classes: 10 digits (0-9) and 4 operators (+, -, *, /). The symbolic program then computes the result of the expression represented by the image. Since Dolphin allows for arbitrary Python objects and functions to be used as symbols and operations, we can easily represent the HWF task as follows:

symbols = [ str(i) for i in range(10) ] + [ '+', '-', '*', '/' ]
res = Distribution(model(img[0]), symbols)

for i in range(1, expr_length):
	op = Distribution(model(img[i]), [ '+', '-', '*', '/' ])
	res = apply(lambda x, y: x + y, res, op)

res = apply(lambda expr: eval(expr), res)
result_logits = get_probabilities(res)

Here, the Distribution associates the logits of the neural model with strings representing the digits and operators. The apply function is used to concatenate these symbols to form strings representing entire expressions. The final result is then evaluated using the eval function. Note that this is a naive version of the HWF task, and the full code for the HWF model is shown in Appendix G.

However, currently, Dolphin does not support non-deterministic symbolic processes. This includes cases where the symbolic program itself may not be known and may need to be approximated. This could be addressed in many ways, such as by supporting weighted operations where the weights themselves are learned during backpropagation, or by even using LLMs to generate the symbolic program. We leave this to future work but will include this as a limitation in the revised manuscript.

Limitations

We mention a limitation of Dolphin in Section 6, namely that programs in Dolphin need to be written in a batched manner, which may pose a challenge for users without experience in deep learning.

Lambda functions

The lambda functions in Dolphin can be any Python function. As a result, the efficiency of the functions itself is dependent on the user. While Dolphin does not provide any acceleration for the lambda functions themselves, the operations on the tags stored in Distributions are already optimized for GPU acceleration. In general, user-defined functions do not pose a significant bottleneck in our benchmarks. To demonstrate this, we present a breakdown of the time required for the lambda functions (referred to as UDFs) and the time required for computing tags in Appendix D.