Automated Discovery of Pairwise Interactions from Unstructured Data

Thank you for your comments on our work. Please find our responses below.

This is an applied paper, and I believe the novelty is weak.

This paper provides the first rigous definition of an interaction between treatment variables on unstructured data & it supports these definitions with two theorems. Further it provides the first active learning approach to image-based outcomes (images as inputs are well-studied) and supports this with rigous experimentation on real world data (that was specifically collected for this study).

Regarding the novelty of our approach, please see our detailed point-by-point response below.

The separability metric essentially measures the mutual information between the latent variables corresponding to a pair of perturbations. There are many different ways to instantiate this, and it's unclear whether the authors' proposed metric itself is a valuable contribution. A similar statement also holds for the disjointness metric.

We respectfully disagree with this point. Section 3 of our work, which introduces the separability and disjointness tests, has two key objectives for unstructured measurements: (1) to rigorously define the notion of interaction, and (2) to design measurable metrics that quantify the intensity of such interactions. To our knowledge, no prior work has addressed these objectives (though we welcome references to relevant literature if we have overlooked any).

We do not claim that performing independence testing via mutual information or additivity testing via learned representations is the key novelty of our work. Instead, our primary contribution lies in rigorously characterizing the types of interactions identifiable by each method and specifying the precise modeling assumptions required for these tests. We believe the theoretical foundation provided by our work offers valuable insights for the community.

Essentially, this paper combines many existing methods to achieve an empirical (and somewhat narrow) goal. I believe this is a valuable contribution, but it may be suited for a more applied venue.

Thank you for recognizing the value of our contribution!

We want to emphasize that this work establishes a general framework for efficiently detecting (potentially rare) pairwise interactions without requiring structured measurements. Given this flexible setting, we believe our methods are broadly applicable across many scientific domains (as outlined in the first paragraph of the introduction). We do not consider this to be a narrow empirical goal.

Can you counter my points (in weaknesses) regarding the lack of novelty in this work?

Please see our previous responses.

The right-hand side of the bottom equation on p.4 seems off; isn't a distribution over a scalar random variable, and the argument is a vector?

If you are referring to the equation in lines 205–207, we believe it is correct. All relevant probability density functions pertain to random variables defined on general measurable spaces $\mathcal{X}$ and $\mathcal{Z}$, which are not required to be scalars.

I think the clarity can be improved (and some space saved) if you apply a log to both sides of Equation (1) and go directly to Equation (3), without writing Equation (2).

Thank you for the suggestion. We will edit the relavent text in the camera-ready to improve clarity.

Thank you again for your comments. We are happy to answer any further questions you have during the discussion period.

I see now that separability is different from mutual information. Seeing that it's the absolute difference between the joint and individual KL terms, there are many ways of computing it besides the approach that you tried (e.g. iVAEs). Since you're presenting a new framework, I think it's important to see the robustness of your conclusions across several implementation methods.

The separability test relies on sample-based KL divergence estimates between the perturbed group and the control group. If the KL estimator is unreliable, the conclusions drawn from the separability test may also lack reliability. However, this does not reflect a lack of robustness in the testing framework itself. The key contribution of this work lies in providing a rigorous framework for understanding what to test, rather than proposing specific KL estimators---a challenging problem in its own right. Our framework is flexible and can accommodate various estimators, allowing users to select an appropriate KL estimation method based on their data type. For example, in our synthetic experiments, a simple KNN-based KL estimator proved reliable for low-dimensional observations.

In our experiments, we used a KL estimation pipeline informed by recent advances in sample-based KL estimation. A detailed explanation of our approach, including the rationale for our chosen pipeline and a survey of alternative strategies from the literature, is provided in Appendix B.

Additional clarifications:

• The reviewer suggested using iVAE (we assume this refers to identifiable VAE; [Khemakhem et al., 2020]) in our framework. However, iVAE does not provide valid KL estimates for samples from two distributions, and we are unsure of its applicability to our approach.
• If the reviewer is proposing latent variable identification (essentially learning causal representations for disentanglement) instead of directly testing the latent dependence from observations, we have discussed in detail in the related work section why this approach is less suitable for our purposes. It's also worth noting that iVAE requires conditionally independent latents (conditional on an environment variable) a prior, while we are tesing whether there is a dependence in the latent space.

I also agree with Reviewer 1aqQ, in that it's important to see your particular definition of interaction compared with some kind of baseline. While I'm not very familiar with this area, it seems implausible that this is the first time anyone has considered the notion of interaction between two vector-valued random variables.

Please see our response to the follow-up comments from Reviewer 1aqQ.

On the right-hand side, I believe $p_{Z_i}$ is a density over a scalar-valued random variable, and its argument $g^{-1}(x)$ is a vector-valued random variable with dimension $L$.

Thank you for pointing this out! This is admittedly an inprecise expression---$p_{Z_i}(g^{-1}(x))$ should be interpreted as $p_{Z_i}([g^{-1}(x)])$ where $$ \cdot $\_{i}$ is the projection operator to the subspace of $Z_i$. The precise expression (updated in the manuscript) is as follows: $\frac{p(x | \delta_i)}{p(x|\delta_0)} = \frac{p_Z(g^{-1}(x)| \delta_i)\left | \text{det}(J(g^{-1}(x))) \right|}{p_Z(g^{-1}(x)|\delta_0)\left | \text{det}(J(g^{-1}(x))) \right|} = \frac{p^\dagger_{Z_i}([g^{-1}(x)]\_i)}{p_{Z_i}([g^{-1}(x)]\_{i})}.$

Thank you for your comments on our work. Please find our responses below.

It is not quantified how much the biological experiments benefit from the active learning algorithm in terms of the total number of necessary perturbations ...

As illustrated in Figure 6, within 50 batches (i.e. 500 pairs of the possible 1225) we are able to discover 12-15% more known interactions than random selection. If one can run all the possible pairs, you would discover all the interaction effects. However, our focus is on efficiently discovering these interactions, which we demonstrate -- 46% interactions discovered in 40% of the total interactions.

I am not sure if it possible but it would make sense to additionally compare the proposed method against causal discovery methods ...

As mentioned in the comment, causal discovery methods are not typically suitable for our problem because it assumes all variables that causally discribes the response are observed, which is not viable in our setting. Our goal is to develop techniques that apply to unstructured data so we do not allow the possibility of "transforming the data into a format that contains a single value for every node".

Causal representation learning is more aligned with our setting. However, as we discussed comprehensively in the related work section, these CRL works all attempt to disentangle unstructured observations into latent variables, and would solve our problem if successful (one could directly observe dependencies among the inferred latents). But they depend on assumptions which are at least as strong as the assumptions that we make here, and if either the assumptions that they make or the estimation procedures they use (i.e. fitting the relevant deep nets) fail, then any conclusions will not be valid. The challenge is that it is very difficult to know whether disentanglement is successful: the assumptions are untestable, and there is no validation set metric that tells you whether you have successfully disentangled latent variables (if there was, then by definition, there would not be an identifiability problem) which makes it very difficult to reliably tune hyperparameters.

By contrast, our interaction tests jointly test both the data generating process assumptions (which you don't want to test) and the independence assumptions (which you do). Our assumptions on the data generating process are comparatively mild, but even if they are incorrect, this will just result in false positives on the test rather than a failure of the inference procedure. Obviously we would prefer to avoid false positives, but in practice, a test that detects real interactions with a high false positive rate is still useful for biologists because it narrows the search space to a relatively small set of candidates.

By pairwise interactions do you mean causal dependencies or correlations? Can you also discover causal relations?

As mentioned in our response to Reviewer 1aqQ, we defined two notions of interactions: violations of separability and disjointedness. While these concepts are related to causal dependencies (particularly the separability), they are not equivalent. Specifically, within the separability framework, interactions are defined as instances where two perturbations target the same latent variables. Similar concepts have been studied in the context of causal representation learning; see [1] for reference.

[1] Score-based Causal Representation Learning: Linear and General Transformations

What do you mean by "unstructured data"? I would say that an image is structured as a grid of pixels.

Unstructured data refers to measurements that do not directly reveal a specific pre-selected property. For example, if we are interested in cell viability, a structured measurement would be the number of healthy cells in a sample, whereas cell images---considered unstructured measurements---require additional processing to extract this information. For a more detailed explanation, please refer to lines 74–84.

Line 094 please elaborate on what you mean "by this notion of interaction".

We rigorously define the two types of perturbation interactions: violations of separability (Definition 3.5) and violations of disjointedness (Definition 3.7).

Line 374 what are 3 dimensional tabular data? Please provide the dimensions.

The data in this example is in $\mathbb{R}^3$. The precise data generating process is provided in Appendix E.1.

Line 478 "Genes from the same pathway are ordered adjacently". Please rephrase/explain what it means.

This means that in the 2x2 matrix in Figure 5, genes from the same biological pathways are placed in adjacent rows (and columns). For the indexing of the selected genes and their corresponding pathways, please refer to Table 1 on page 24.

Thank you for the comments on our work. Please find our responses below.

The two proposed interaction tests are not compared with any standard methods ...

We thank the reviewer for these suggestions and will include a discussion in the camera-ready version. However, the first two classical methods from Cox and his coauthors do not apply in our setting because they focus scalar outcome variables whereas the primary motivation for our paper is to ask how to generalize a notion of interaction to unstructured outcome data like the pixels in an image. Our task is significantly more challenging because we not only have to contend with the high dimensionality of image data, but our notion of interaction has to be robust to arbitary bijective transformations of the data (because we cannot assume that we are in control of measurement process). To emphesize this point, consider this quote from Berrington de González and Cox [2007] which reveals just how dependent the classical approaches to interaction testing are on the judgement of a human analyst, "For a continuous and positive response variable, $y$, the transformations commonly used are logarithmic and simple powers, occasionally with a translated origin. For binary data, the logistic or sometimes probit or complementary log scale may be effective. While achieving additivity of effects is helpful, interpretability is the overriding concern. Thus, the transformation from $y$ to $y^{1/3}$ might remove an interaction but, unless y was a representation of a volume, $y^{1/3}$ might well not be a good basis for interpretation." Please refer to our response in the relevant section for further details.

The third reference from Akimov [2021] serves as a fantastic motivation for the problem that we solve in this paper when they argue that, "emerging phenotypic profiling methodologies will improve the discovery of therapeutically relevant, novel SL interactions. [emphasis added]". They emphasize the need for more general notions of synthetic lethality (e.g. "synthetic sickness"; similar to your more general point about epistasis below), but they do not provide methods for detecting these more general notions of synthetic lethality (their Figure 4 D appears to suggest using tSNE as a way of identifying relevant outcome variables; this will lead to very unreliable tests!). Our interaction tests solve exactly this problem.

Biologically, this issue is known as "epistasis," a well-researched topic with substantial existing ....

Your point that "domain knowledge and modeling of underlying structures are essential" is likely the key point of where we disagree in the value of this paper. Domain knowledge is clearly sufficient if it allows one to extract a well-defined outcome variable from unstructured data. But neither of the referenced papers claim it is necessary to detect interactions. In this paper we give two well-defined notions of interaction that apply to unstructured data. Because they do not depend on domain knowledge, they will tend to have lower specificity than tests for known interaction targets; but this also allows them to be far more scalable in detecting interactions in high-throughput screens. If every gene pair that interacted produced a different morphological phenotype, then you'd need a different test for each phenotype; whereas our tests allow you to detect interactions without prespecifying the form of the interaction.

We concede that we have done a poor job of connecting with the biology literature, and we will update our references accordingly; thank you for pointing these out. That said, we emphasize that the key contribution of this work is in the interaction testing methodology itself, which we then illustrate on biological data; this is not a biology paper and does not claim to be. This is an ICLR submission: the original focus of this conference has always been on general methods for learning from unstructured data, so tests that allow us to move beyond assuming access to domain knowledge are very much the core methodological focus of the conference.

Thank you for your detailed response. I believe I now understand the main points of disagreement, but let me further make sure three key points as below.

Rating wise, since we are reviewing the submitted manuscript, not any unseen revised versions, I keep the scores as is.

(1) Lack of comparisons with other methods

The response to this issue remains unconvincing. I do not believe it is impossible to compare your method with others.

While your paper refers to “unstructured data,” it ultimately converts image features into embedded vectors using a pre-trained model—essentially transforming them into fixed-dimensional multivariate data—which are then used for interaction testing. As other reviewers may have pointed out, statistical interaction tests can be applied to multivariate data. Hence, a lack of comparisons with alternative methods is not justified.

(2) Identifying pairs with interaction effects “without directly measuring pairwise effects”

I guess that the “Automated discovery” in the title refers to the explanation in L95 of the paper: "We can search the space of pairwise experiments by selecting pairs of perturbations that are likely to result in large test statistics. In doing so, we reduce the problem of finding interacting pairs of perturbations into an active matrix completion problem."

If I understand correctly, this means predicting test statistics for unmeasured pairs (k, l) based on observed perturbation results for other pairs (i, j). This would imply predicting pairwise effects for (k, l) without directly measuring their double perturbation effects, wouldn't it? This is the point I find the most difficult to fully accept regarding the validity of the setup.

From the author response, I guess that this is possible due to assumptions employed in an existing work [1], such as the matrix being low-rank and columns following a Gaussian distribution. These assumptions are briefly mentioned in L349 of your manuscript. But [1] would not specifically target the one of this paper, i.e., pairwise interactions like synthetic lethality, simply relying on existing assumptions should require careful validation and discussion within the paper. Are these assumptions not something that can be derived from Assumptions 3.1, 3.2, and 3.4, but rather supported only by the empirical fact that the experimental results seem to work well? The paper feels theoretically solid in parts, but the critical sections come across as very loose.

This gives the impression that this paper just borrows the established framework [1] for "automated discovery" without fully assessing whether its technical assumptions (e.g., low-rank matrix structure, Gaussian-distributed columns) are appropriate for this paper's research goal. From its definition, the pairwise effects this paper targets are seemingly hard to predict by partial observations of other pairs, and thus careful validation and discussion of these assumptions seem necessary.

[1] Adaptive sampling for discovery. NeurIPS 2022, 35, 1114–1126.

(3) "domain knowledge and modeling of underlying structures are essential"

My comment on this point might have been confusing. I did not mean to refer to any conclusions from a specific study. Rather, I meant that prior research on detecting epistasis (or synthetic lethality as a specific case) often incorporates additional information, such as PPI, GO, pathway data, or knowledge graphs. See the following, for example.

• [2] Benchmarking machine learning methods for synthetic lethality prediction in cancer. Nat Commun 15, 9058 (2024). https://doi.org/10.1038/s41467-024-52900-7
• [3] Discovery of synthetic lethal interactions from large-scale pan-cancer perturbation screens. Nat Commun 13, 7748 (2022). https://doi.org/10.1038/s41467-022-35378-z

This is because from the definition, synthetic lethality is hard to identify only from the indirect observations. Predictions therefore usually require additional assumptions or information.

In this sense, the additional assumptions in this paper would be the one from [1], i.e. low-rank matrix structure, Gaussian-distributed columns. But this point should have been more carefully verified and discussed in the paper. Plus, further validation, discussion, and comparisons with existing methods seem necessary. For reference, [2] includes benchmarking of three matrix factorization-based methods (SL2MF[4], CMFW[5], GRSMF[6]).

• [4] SL2MF: Predicting synthetic lethality in human cancers via logistic matrix factorization. IEEE/ACM Trans. Comput. Biol. Bioinform. 17, 748–757 (2020).
• [5] Predicting synthetic lethal interactions using heterogeneous data sources. Bioinformatics 36, 2209–2216 (2020).
• [6] Predicting synthetic lethal interactions in human cancers using graph-regularized self-representative matrix factorization. BMC Bioinform. 20, 1–8 (2019).

We thank you for the helpful feedback on our work. Please find our responses below.

The use of a statistical test for matrix completion is a little poorly motivated. In particular, given an n x n matrix completion problem, at least one pair is likely to have a large test statistic ...

This is a great question and worth discussing in more detail. Our original motivation for working on this problem was thinking about exploring, e.g. all pairwise knockouts of the human genome (approx. 20 000 genes, so at least 200 million experiments). In spaces this large, you're always bounded by budgets and so optimal stopping is not a concern. That said, we have followup work to this which considers the problem of how to make single perturbation experimentation more efficient [no citation; currently under review]. In both this paper and that work, our acquisition function essentially optimizes for examples where we are likely to make incorrect predictions (under the disjointness statistic; see section 3.2). Because of this, if our posterior over $\mathbf{R}$ is calibrated (this can be relaxed to weaker notions of calibration), then for each batch of samples, $B$, that you select at each round, the average loss over the batch, $\frac{1}{|B|}\sum_{i,j \in B}\|\vec{h}_{i,j} - (\vec{h}_i +\vec{h}_j)\|$ is a upper bound on the average loss of IID samples from $\mathbf{R}$, and hence can be used to construct a explicit stopping criterion. Intuitively, this can be thought of as, "if you're accurately predicting all the pairs you think are most likely to interact, then there probably are no interactions left in the matrix." For this approach to work, you need large enough batch sizes, but in practice batch sizes tend to be large to keep experimental costs per sample low.

Second, I think the authors did not explain why one would use imaging data for this matrix completion problem, instead of the more structured RNAseq data.

As explained in lines 74–84, unstructured measurements, such as cell microscopy images, can be acquired through automated high-throughput perturbation platforms, making them a significantly cheaper alternative to structured measurements like RNA-seq. While cell images may provide less precise signals for a specific pre-selected properties, they retain rich biological information [see Celik et al. 2024 for a comparision showing the similarity between RNA-seq and microscopy data]. The challenge, of course, is that we need methods that are able to work with unstructured data directly (rather than rely on preprocessing or specially designed assays). To our knowledge, this is the first work the provides a rigorous approach to achieving this goal of detecting interactions from unstructured data. As such, we view the interaction tests as our primary contribution. We performed inference on these unstructured measurements (microscopy images) to demonstrate that these test work even in the extremely challenging real world setting of microscopy imaging.

Third, I did not get a sense of how often the joint distribution differs from the product of the marginals in this setting...

Thank you for bringing up this very relevant work. Ahlmann-Eltze et al [2024] is concurrent work (note that it was published the week before the ICLR abstract deadline), but they arrive at a very similar conclusion to our original experiments that motivated our disjointness test. The additive model that outperforms the deep learning based approaches in Ahlmann-Eltze et al [2024] is, $x_{i,j} = x_i + x_j − x_{0}$ (see equation 2 in [Ahlmann-Eltze 2024]). Because they are working with bulk expression data, this should be interpreted as $E[x|\delta_{i,j}] = E[x|\delta_i] + E[x|\delta_j] − E[x|\delta_{0}]$ in our notation (i.e. $x_i$ corresponds to the average expression from knockout $i$). This is just a special case of equation 5 in our paper, when you choose the embedding function $h(\cdot)$ as the identity function. Thus, our disjointness test can be interpreted as a test for when the additive model in Ahlmann-Eltze [2024] will provide good predictions of double knockouts. It is also much more general, because it also provides sufficient conditions for additivity for any embedding function, $h(\cdot)$. We use this observation as a starting point to our method: rather than try to predict double perturbation outcomes (which as Ahlmann-Eltze [2024] point out, has been fairly unsuccessful thus far), we aim to experiment only where this assumption fails. The heat maps in Figure 5 left and middle show two different estimates of where it fails. Consistent with Ahlmann-Eltze [2024], there is a lot of "dark blue" in the matrix where the additive model would work well, but there are also many places where it fails.

[EDIT: just saw the note to all reviewers about working on a new pdf — no rush, thank you!]

Thank you for your extensive response. I am still slightly confused about what you mean by a "calibrated" posterior over the matrix $\mathbf{R}$. The typical notion of calibration — which you gave — is a measure of how far P(Y=y|f(x) = p) typically is from p. As you note, this is measured empirically in classification settings. However, in your setting $\mathbf{R}$ is a continuous density, so I am unsure what you mean by a "calibrated" posterior in the language of, e.g., Guo 2017. You seem to hint at it with your comment on "outputs a discrete distribution over a sufficiently fine gained binning of rewards even though they are really continuous." Could you please clarify this comment? I understand the general intuition, however, that calibration is a weak condition and that a calibrated model could be inaccurate on individual instances, so calibration doesn't necessarily imply stopping at step 0.

The second point is interesting. However, I am unable to see any appendix figure in the PDF mirroring Fig. 5. I believe the PDFs can be updated until tomorrow — if a PDF with a fresh copy of this Figure is uploaded soon and convincingly makes the case that the additive model is insufficient in large regions of the matrix, I would be willing to raise my score.

Dear Reviewers,

We thank you again for the engaging discussions and thoughtful feedback on our work. We have now updated our submission to reflect the dicussions.