Thank you for your valuable comments and opinions.

Weaknesses

We understand that the notation is a bit heavy, as it is hard to combine the notations of causal inference and object-oriented representation. We will try to improve the clarity of the notations in the revised version.

If we apply identification encoding (each object has an attribute $O.\textrm{ID}$), just like position encoding for Transformer, then Causation Symmetry usually holds.

It's true that OOFULL outperforms CDMs. There are 2 main factors that lead to overfitting:

1. Making predictions from spurious correlations. CDMs are able to ameliorate this factor by learning the causal structure.
2. Even with oracle causal discovery, overfitting can still occur within the structural equations because they are still learned from a limited amount of data representing the entire space. CDMs have no capacity to solve this problem, which explains why CDMs fail on o.o.d. data. For example, CDL-A performs better than CDL at causal discovery, but generalizes worse than CDL. The reason is that the attention in CDL-A is much stronger than the pooling mechanism in CDL, leading to more overfitting within the structural equations.

One way to reduce the overfitting from the second reason is to increase the number of samples. OOCDM and OOFULL are able to do this because each instance of the class $C$ is treated as a sample for learning the predictor $f_{C.V}$. Thus, the sample size increases significantly. Another reason for OOFULL to generalize better is that causation symmetry limits the VC dimension of the model, which reduces the chance of overfitting.

Based on the above analysis, the generalization ability of OOCDM benefits from 3 sources: causal structure, causation symmetry, and increased sample size. OOFULL benefits from causation symmetry and increased sample size, but not from causal structure. CDMs, on the other hand, only benefits from causal structures. As a result, we observe that the performance is OOCDM > OOFULL > CDMs.

GRADER indeed suggests using online learning to improve its performance. In this paper, we do not specify how OOCDM should be used, and online learning is absolutely acceptable. We experimented with offline data because it is the best way to demonstrate the generalization performance of the model. If OOCDM can handle this challenging case, we have no doubt that it can perform better when online data is available. Moreover, GRADER's theory requires that we repeatedly perform causal discovery with new data. Considering that the causal discovery of GRADER can take hours, we doubt that it is practical to make comparisons with CDM baselines.

Questions

The number of objects does not vary along the same trajectory. Otherwise, we may need to predict when a new object enters the environment and when an existing object leaves the environment (we leave this to future work). However, the number may vary between episodes. For comparison with baselines in Sections 5.2, 5.3, and 5.4, we fixed the number of objects so that baselines can be applied, and the performance of different numbers of objects are reported separately (Block2, Block5, and Block10). In Section 5.5, when we tested the performance of OO models under different number of objects, we only compared with OOFULL, because other baselines do not apply.

The causal graph accuracy is measured by SHD. We will include this in the main paper.

GRADER employs Fast CIT (FCIT), for which we use the official FCIT package from github. To test whether $X$ and $Y$ are independent conditional on $Z$, the idea of FCIT is to train 2 decision trees -- one predicting $Y$ using only $Z$, and the other predicting $Y$ using both $X$ and $Z$. Then a t-test is performed based on the mean square errors of the two decision trees. To ensure that the test is "fast", the time to train these trees is quite limited, so the precision of the trees is worse than the neural networks used in CDL and OOCDM. It is the low precision of these decision trees that compromises the performance of causal discovery in GRADER.

About minor problems

Sorry for the typos. We will correct them in the revision.

The expression $C_k[C_k.U \rightarrow V]$ is different from $C_k[U \rightarrow V]$. Assume that $C_k$ is $product$, where $U$ is $price$, and $V$ is $sell$. Then, $Product[Price \rightarrow Sell]$ means that the sell of each product is affected by its own price. $Product[Product.Price \rightarrow Sell]$ means that the sell of each product is influenced by the price of all other products (which is true in a competitive market). Note that $k = l$ is not forbidden in Definition 2. That's why we have to distinguish between notations for local and global casalities. Maybe $C.(U \rightarrow V)$ and $C_k.U \rightarrow C_l.V$ is better here?

We will include adjacency matrices in the appendix.

A fully-connected bipartite graph satisfies the definition of an OOCG, where all causality expressions hold.

Dear reviewer, we have submitted the revised version of our paper. These changes in the revision may be related to your concerns.

1. In some cases the bold letters (e.g. $\mathbf{S}$) are not distinguishable from the normal letters (e.g. $\mathrm{S}$). Therefore, now a normal letter always carries a subscript (e.g., $\mathrm{A}_i, \mathrm{S}_i$, $\mathrm{S}_j$) so they can be well distinguished.
2. The notation $\mathbf{U}$ is no longer used. Instead we will use $(\mathbf{S}, \mathbf{A})$ directly. This way we avoid introducing unfamiliar notations, and $\mathbf{U}$ will not be confused with the letter used in a field $C.U$ or an attribute $O.\mathrm{U}$.
3. We have changed the notation of local class-level causality to $C.U \rightarrow V$, and that of glocal class-level causality to $C_l.U \rightarrow C_k.V$. These notations are simpler and more intuitive.
4. We simplified equations (5-10) in the old version, and now there are only two equations (5-6) in the revised version to express the same meaning. This reduces the reader's workload and the complexity of the notation.
5. The subscripts of objects in Eq. 4 (now Eq. 3) are now corrected.
6. in Table 7, "action" is corrected to "state" for the meaning of $O.\mathbf{S}$.
7. We now write a field predictor using the form $f_{C.V}(O.\mathrm{V}'|\mathbf{S},\mathbf{A};\mathcal{G})$.
8. In Section 5.2, we have explained that the accuracy of causal graphs is measured by Structural Hamming Distance.
9. We have visualized the adjacency matrices of the ground truth OOCGs in Appendix G and the discovered OOCGs in Appendix H. These figures are intended to be more direct and understandable than the old presentation of OOCGs.
10. In Appendix I, we have provided a simple solution (with an additional experiment) to extend OOCDM to environments where causation symmetry does not hold. In short, by adding auxiliary identity attributes to the objects, we can ensure causation symmetry and result symmetry in all OOMDPs.

Thank you for your response.

We limited the computation of accuracy within the edges that a bipartite CDM needs to identify. After causal discovery (both OOCDM and CDM), we represent the causal graph by the adjacency matrix from $(\mathbf{S}, \mathbf{A})$ to $\mathrm{S}_j'$. Then we measure the differences between the true and estimated matrices.

Suppose $\hat{M}_{n_s \times (n_s + n_a)}$ is the estimated adjacency matrix.

Assume that $M_{n_s \times (n_s + n_a)}$ is the true adjacency matrix.

Here, $M_{ji} = 1$ means that the $i$-th variable in $(\mathbf{S}, \mathbf{A})$ is the parent of $\mathrm{S}_j'$.

We count the number of different elements in the two matrices. That is, if $\hat{M}_{ji} \neq {M}_{ji}$, an error is counted. Therefore, each missing edge or redundant edge is considered as an error. Then we express the accuracy as a percentage:

$\frac{n_{correct}}{n_s \times (n_s + n_a)} \times 100$.

We hope this addresses your doubt, and we will be grateful if the score can be adjusted accordingly.

Thank you for your comments.

About weaknesses

First of all, we understand your concern about the readability of the paper. We will try our best to improve this in the revised version.

Weakness 1

Regarding notations, you can find a list of all notations in Appendix A. Combining the notations for object-oriented representation and causal inference was more complicated than we thought, so there may be some confusion. We use recommended notations in the ICLR template, but it's true that bold and normal letters are sometimes indistinguishable. We will try to find a way. How about $\vec{\textrm{S}}, O.\vec{\textrm{S}}$?

Currently we are considering the FMDP setting. If the transition depends on time, just make "time" a variable (which the predictor is manually given as $t' = t + 1$) in the FMDP.

Weakness 2

$\mathcal{F}$ and $\mathcal{F}_s$ are explained in Section 3.2:

• "$C_k$ specifies a set $\mathcal{F}[C_k]$ of fields that determine the attributes of $O_i$ as well as other instances of $C_k$."
• We will use $\mathcal{F}_s[C_k]$ to denote the set of state fields of $C_k$ corresponding to state attributes.

Since Definition 2. is a bit far from Section 3.2, it's hard for the reader to catch up. We will add a sentence after Definition 2 about where to find the meaning of $\mathcal{F}$ and $\mathcal{F}_s$.

Weakness 3

This is a good question. In our model, we can capture the different influences of classes by using class-specific encoders. In other words, each class has its own encoders to encode key and value vectors. See Appendix E for more details. For the predictor $f_{C.V}$, the class uses a key encoder $KEnc_{C_k \rightarrow C.V}$ and a value encoder $VEnc_{C_k \rightarrow C.V}$ to compute the key and value vectors. These class-specific encoders account for the unique mechanism of each class in the dynamics.

Weakness 4

We accept your suggestion to briefly explain CIT in the paper. Equation (12) is used to train the predictors in the model instead of learning the causal graph. Then the predictors are used to perform causal discovery through equations (6, 7, 8, 9, 10). Thus, the learning of the causal graph itself is not directly based on gradients. The pseudocode in Appendices E.2. and E.3. further clarifies this process.

Questions

Question 1

The distribution is determined by the domain of the variable, and the domain is given. If the domain is discrete and finite, the distribution is categorical, where the predictor outputs the probability of each category. Otherwise, we assume the distribution is Gaussian and the predictor returns the mean and standard variance.

Question 2

We accept your suggestion about this question and will make the change in the revision.

Question 3

Sorry for the misunderstanding. The symbol (without /) means conditional independence, and the negative (used in eq. 2) means conditional dependence. We will correct this in the revision.

Question 4

Consider testing an independence relation between $X$ and $Y$, conditional on $Z$. Typically, CIT methods learn two prediction models $\hat{Y} = f(Z)$ and $\hat{Y} = g(X, Z)$ -- that is, we try to predict $Y$ with and without $X$. If the independence holds, $Y$ does not carry information from $X$, so the two prediction models should yield identical results; by comparing the difference between the two prediction models, we can determine whether independence holds. The computational complexity of $f$ and $g$ is bounded by the size of their inputs, which are $O(n)$ in equation (2). Therefore, checking each edge costs at least $O(n)$.

Question 5

(a) Sorry for the typo. An ellipsis is missing after $O_{i+1}$. We will correct this in the revision.

(b) We use the semicolon in equation (3) to emphasize the order of the condition variables to which function $p_{C}$ may be sensitive. For example, assuming that $O_1$ and $O_2$ are objects of the same class $C$, based on (3) we have

$p(O_1.\textrm{S}' | \textrm{U}) = p_{C}(O_1.\textrm{S}' | \textrm{O}_1; \textrm{O}_2, \textrm{O}_3, \cdots, \textrm{O}_N)$

$p(O_2.\textrm{S}' | \textrm{U}) = p_{C}(O_2.\textrm{S}' | \textrm{O}_2; \textrm{O}_1, \textrm{O}_3, \cdots, \textrm{O}_N)$

As we can see, the order of the condition variables in $p_C$ is different in the two equations. Thus, the way $p_C$ handles $\textrm{O}_1$ for $O_1.\textrm{S}'$ is the same as the way it handles $\textrm{O}_2$ for $O_2.\textrm{S}'$. The semicolon can emphasize this positional meaning of condition variables, so it is clear that $\textrm{O}_i$ always takes the first position. Otherwise we would have $p(O_1.\textrm{S}' | \textrm{U}) = p(O_2.\textrm{S}' | \textrm{U})$, which is incorrect.

Dear reviewer, we have submitted the revised version of our paper. These changes in the revision may be related to your concerns.

1. In some cases the bold letters (e.g. $\mathbf{S}$) are not distinguishable from the normal letters (e.g. $\mathrm{S}$). Therefore, now a normal letter always carries a subscript (e.g., $\mathrm{A}_i, \mathrm{S}_i$, $\mathrm{S}_j$) so they can be well distinguished.
2. The notation $\mathbf{U}$ is no longer used. Instead we will use $(\mathbf{S}, \mathbf{A})$ directly. In this way we avoid introducing unfamiliar notations, and $\mathbf{U}$ will not be confused with the letter used in a field $C.U$ or an attribute $O.\mathrm{U}$.
3. We have moved the explanation of $\mathcal{F}_s$ into Definition 2 where the notation is used.
4. We have added a brief explanation of the Conditional Independence Test (CIT) in Section 3.1. Meanwhile, more information about CIT and how CMI is used for CIT is added in Appendix B.4.
5. The notation of conditional independence in Theorem 1 is now correctly explained. We explicitly use "$\neg$" to avoid confusion.
6. We have added the mathematical expression of "independent transition" in Theorem 1.
7. We have added the missing ellipsis in Eq. 3 (which is Eq. 2 in the revised version).
8. We have added the recommended papers into related works.

Thank you for your response.

We have proposed to introduce "instance ID" to handle the unique property of objects in asymmetric environments. You suggested "class ID" for objects and using the common predictors for all objects, which may work in theory ( provided all classes share the same set of fields). However, we believe that modelling multiple classes has several advantages:

1. Different classes can have different sets of fields. For example, the class "Human" has a field "Gender", but the class "House" has no such field. By modelling classes, the OO representation is more flexible to such differences.
2. The Class ID solution can be problematic in practice. The instinct behind it is that "objects from the same class should be similar, but objects from different classes are usually very different", which is a natural concept when people represent the world using the OO technique. Therefore, although instances of the same class may have different dynamics, the difference should be small, so using "instance ID" allows the model to infer the difference. However, since the dynamic discrepancy between classes is much larger than that between instances of a class, it would be difficult to infer such a discrepancy from class ID using a single neural network.
3. Finally, the use of class ID can severely compromise causal discovery. For example, suppose there are two classes, $Male$ and $Female$, and they have the same set of fields. If we model the two classes differently, we may discover unique causality for each class, e.g. $Female.Beauty \rightarrow Wealth$ but not $Male.Beauty \not\rightarrow Wealth$ (just an example here). However, if we do not treat them differently, we will have $Object.Beauty \rightarrow Wealth$ and $Object.Gender \rightarrow Wealth$ (in this case, gender is the class ID). Then the causal graph does not say that the causality only exists for women. This problem would be much more severe if the difference between classes was large. In fact, the same problem exists in our instance ID solution. However, because objects in the same class are very similar, the problem is tolerable.

It is important to note that humans naturally categorize objects in a way that minimizes the differences within a class and maximizes the differences between classes. This motivates us to model different classes and benefits the performance of OOCDM. The same idea is used in modern object-oriented programming. For example, Python also uses "obj.class" as a "class ID", but it also benefits from modeling class-level behavior.

Unfortunately, no further experiments can be done as the rebuttal phase is about to end. If you are still interested, you can leave your email after the final decision and I will send you the results.

We hope this answer clears up your doubts. We would be grateful if the score could be adjusted accordingly.

Thank you for your kind opinions and comments.

Weakness 1

In this paper, we focus on Factored MDPs (FMDPs), so that methods based on FMDPs would benefit from our work. The baselines (CDL, GRADER, TICSA) are all based on the settings of Factored MDP, and their settings also exclude confounders. Thus, we only claim to provide a solution to the current scalability problems of these baselines. With respect to the claim of our paper, we believe that using FMDP does not harm the rigor of this paper.

We agree that the use of FMDP imposes limitations. As you pointed out, there are no confounders and the transition is strictly Markovian, which is not applicable in some real-world problems. We will add this statement about this weakness in Appendix I. Extending our methods to an environment with non-Markovian dynamics, partial observability, or latent confounders is interesting but also challenging. In our future work, we will certainly investigate such more challenging problems.

Weakness 2

The reason why [1] is missing in Section 3.2 is that the definition of [1] formulates the dynamics through a set of logical rules (so [1] is a special case for our problem). We do not hope to impose this strong restriction on our setting of the problem, so we avoid citing [1] in Section 3.2. Since "OOMDP" seems to be the most relevant way to name the problem, we have also used it in our paper. We will add a brief explanation of this issue in Section 3.2 in the revision.

Weakness 3

The problem can be solved by introducing additional attributes in the representation. We can include "identity encoding" in the attributes of objects. For example, we can define $Obj.ID$ fields for these objects. For example $Obj1.\textrm{ID} = 1, Obj2.\textrm{ID}=2, ...$ Then the objects become interchangeable, so causal symmetry holds. Then these IDs will be well handled by our attention-based network, which can naturally assign high attention weights to connected objects.

Weakness 4

You are right. The key of our method is to share casual structures between objects. But if all objects are from different classes, there is no way to share causal structure. In this case, we have $m$ (the number of fields) equal to $n$ (the number of variables), then the time complexity of causal discovery becomes $O(n^2N)$, where $N$ is the number of objects. In the worst case, we have $N = n$ (each object contains only 1 variable), then the complexity is as bad as for non-object-oriented methods. However, we hope to clarify that our method is not designed to solve such an extreme case, which is against the motivation of our approach.

Weakness 5 and question 1

The baseline CDMs (GRADER, CDL, TICSA) are all based on factored spaces, so adopting FMDPs in our settings is reasonable for our claim. In fact, we came up with the idea of this paper when trying to learn CDMs in some previous studies, where we found that causal discovery is really slow, and a way to speed up the process is in dire need even in factored environments.

There are studies on object-centric representations (OCR), which can learn homogeneous object representations from image inputs. Thus, OCR methods are compatible with our approach when the number of classes is exactly 1, and some adaptation of the networks and the loss function may be required. However, we believe that the solution should not be limited to single class cases, which does not fully exploit the potential of an OOCDM.

In future work, we may try to extend OCR to multi-class scenarios (e.g., a one-hot vector in the feature of each object to identify its class, and then use different decoders for each class to obtain the attributes). However, much deeper work is needed to ensure that the representation follows causal symmetry in the future.

Question 2

We tried to give a more rigorous definition in Appendix D.1. (Please ignore some minor typos there, as the submission was in a hurry.)

Question 3

In fact, the paper in ICML 22' (which we refer to as "CDL") is one of my baselines in the experiments. The original CDL uses a pooling mechanism to aggregate information from causal parents, which we found to be very ineffective in large cases. Therefore, we also considered the attention-based variant of CDL, called "CDL-A". Here are the general conclusions of the comparison:

1. The accuracy of causal graphs (Table 1): Ours > CDL-A > CDL
2. The accuracy of prediction (Table 3): In the training set, we have Ours > CDL-A > CDL. However, in the in-distribution test set and out-of-distribution test set we have Ours > CDL > CDL-A, so it seems that attention causes more overfitting for CDL.
3. Computational complexity (Table 2): Theoretically, Ours <= CDL = CDL-A. Experimentally, Ours <= CDL < CDL-A. Here, Ours = CDL happens only in small-scale environments.
4. CDL and CDL-A cannot adapt to varying numbers of objects, but ours can.

In summary, our model is certainly much more efficient than CDL and CDL-A.

Dear reviewer, we have submitted the revised version of our paper. These changes in the revision may be related to your concerns.

1. In Appendix J (Weaknesses and future works) we have added a discussion about the limitations of using the Factored MDP. In the future, we will explore more complicated scenarios involving partial observability, confounders, and non-Markovian dynamics.
2. In Section 3.2, we have added a sentence explaining why our OOMDP does not fully conform to the definition in "An object-oriented representation for efficient reinforcement learning" (Diuk et al).
3. In Appendix I, we have provided a simple solution (with an additional experiment) to extend OOCDM to environments where causation symmetry does not hold. In short, by adding auxiliary identity attributes to the objects, we can ensure causation symmetry and result symmetry in all OOMDPs.

Thank you for your reply.

About Weakness 3

We have provided a formal proof in Appendix I.1. In short, no matter how we change the order of the objects, the unique IDs allow us to restore the original order. In this way, OOCDM can learn to map the ID to the unique property of the object.

For example, we have $O_i.ID = i$ in the system, and use $O_i.X$ to denote the old attributes of $O_i$ without the ID. Then, in the first scenario you described, we are interested in $p(O_1 | O_2.Force=1, O_2.Id=2, O_3.Force=0, O_3.Id=3)$ In the second scenario, we are interested in $p(O_1 | O_2.Force=0, O_2.Id=2, O_3.Force=1, O_3.Id=3)$ Checking the definition of causation symmetry, the model does not consider them strictly symmetric, because the $ID$ attributes are not swapped between O_2 and O_3. By inserting IDs, causation symmetry should be: $p(O_1 | O_2.Force=1, O_2.Id=2, O_3.Force=0, O_3.Id=3) = p(O_1 | O_2.Force=0, O_2.Id=3, O_3.Force=1, O_3.Id=2),$ where the IDs of $O_2$ and $O_3$ are also swapped. Then the OOCDM can easily restore the original order of the objects. In our implementation, the information of the IDs is encoded in the key, value, and query vectors in the attention module, which successfully handles the IDs in the additional experiment.

You can think of the IDs as the "positional encoding" in the Transformer architecture. The key-value attention mechanism is agnostic to the order of the tokens, but the positional encodings allow Transformer to infer spatial information.

About Weakness 4

It is true that the proposed method requires manual construction of the OOMDP representation. Therefore, we only claimed to learn causal dynamics in object-oriented environments instead of arbitrary environments. Piror's work [1,2], which studies the same problem, is also based on given representations, and we do not claim to remove this requirement. Constructing the OOMDPs from pixel input or raw factorization is still an open problem, and we will certainly investigate this direction in the future. There are potential solutions, such as extending OCR to multi-class domains. However, proposing any efficient solution is a huge breakthrough in the field and enough to support another paper.

In addition, the solution of ID attributes for objects allows causation symmetry and result symmetry to be violated. OOCDM still works well if the violation is not severe. In this way, constructing the OOMDP representation is flexible, as it's not necessary to guarantee the result and causation symmetries.

If there are many classes, we can also combine similar classes into a superclass. An auxiliary field specifying the subclass can be inserted into the superclass so that the model can infer the subtle differences of objects in these combined classes. For example, creatures in our world come from countless species, and creating a class for each species is indeed cumbersome. However, we can categorize creatures according to a coarser standard and simply use superclasses like $Animal$ and $Plant$, and the species of the creature is described by an additional attribute $O.\mathrm{Species}$. In our future work we will also investigate more complicated class hierachy, so that the model can better handle scenarios with many classes.

Moreover, we believe that even though the OOCDM requires a known representation, it is still useful:

1. There are many environments that are inherently object-oriented. Like StarCraft, every unit is an object. In some multi-agent domains, such as multi-agent partial environments (MPE), each particle is an object.
2. Describing the environment in the object-oriented way is quite natural, so that the representation can be constructed from the instinctive understanding of the users (i.e. we don't need a real expert).

[1] C. Diuk, A. Cohen, and M. L. Littman, “An object-oriented representation for efficient reinforcement learning,” in Proceedings of the 25th international conference on Machine learning, 2008. [2] C. Guestrin, D. Koller, C. Gearhart, and N. Kanodia, “Generalizing plans to new environments in relational MDPs,” in Proceedings of the 18th International Joint Conference on Artificial Intelligence, 2003.

I sincerely hope these answers adress your concerns, so that the scores can be adjusted accordingly.

Thanks for the response.

First of all, in theory, if we provide the ID attributes, there is always an OOCDM that represents the dynamics. This is formally proven in the appendix. However, learning this OOCDM may not be tractable if the violation is severe, because the OOCDM is implemented by neural networks with a limited number of parameters and must be trained with a limited amount of data.

Therefore, "the violation is not severe" means that this dynamics is within the capability of a practical neural network -- that is, the mapping from the IDs to the individual influence on the dynamics is trainable for our attention-based networks. Otherwise, we might need a really deep network and a large amount of data to learn this dynamic. For example, in an extreme case, objects of the same class choose completely different dynamics functions based on their IDs, and these functions do not share any common factors. Then the OOCDM has to remember each function independently and learn to choose the correct function based on the ID. In this case, the violation is so strong that the discrepancy between objects of the class is not learnable.

Conceptually, we can write the dynamics for an object $O_j$ as $p(O_j|O_1,...,O_N) = f \big( g(O_1^{-ID},...,O_N^{-ID}), h(O_1,...,O_N), Noise \big)$ where $g$ is the shared and interchangeable part of the dynamics, $h$ involves the individual influence of the objects, and $f$ merges the shared and individual influence into the final dynamics. The "large violation" then means that $h$ has a dominant effect and is too complicated to learn. Note that the above fomulation is just an example. The true dynamics may take another form, but it is supposed to have shared and individual parts.

Fortunately, humans naturally categorize objects in a way that minimizes differences within a class and maximizes differences between classes. Therefore, even though objects may have individual properties, these properties are easily inferred from their IDs. Therefore, the proposed OOCDM is suitable for most cases because the mapping from IDs to the unique properties of objects is usually learnable.

There is no rigorous mathematical definition of "severe violation", since the implementation of OOCDM can always be augmented with more powerful networks so that the dynamics can always be learned with sufficient data (in theory, neural networks can be arbitrarily strong). In this work, we may do so by increasing the layers of key, value, and query encoders in the predictor networks. However, we are talking about an affordable implementation of OOCDM with high computational efficiency -- if such an implementation exists, we say "OOCDM works well", which happens when the mapping from IDs to the individual influence of objects is not difficult to learn.

Thank you for your suggestions and comments. It is a great luck to see you again as my reviewers, and the suggestions on the previous version really helped me in improving this work.

About Weakness 1

The suggestion is very useful here. In fact, In Section 4.1 we write "The local causality describes shared structures within each object" and used Figure 3(a) to further exlain the concept. We will add a footnote with citation to [1] in the revision to reduce any misunderstanding.

About Weakness 2

It is a interesting topic to consider "relational interactions" in object-oriented dynamics. Here we may come up with a few solutions for that concern.

First of all, our mathod supports an implicit solution. We may assign an additional identification attribute called "ID" for each object, then the attention model should be able to capture the relational interaction. High attention weights will be assigned to the objects that take part in the interaction by matching to their IDs. If there are different kinds of relations, we may use multi-head attention in the model. The drawback of this approach is that these relations are implicit in the causal graph.

The second solution is introducing additional attributes about who the object may be interacting with. For example, in the starcraft environment, we may add a feild like "Marine.AttackTargetID", which gives the ID of the target object that the marine is attacking in an "attack" relation. A special value for the attribute means that the marine is attacking no one (i.e., not in an attack relation). Then, we probably will have these causations in the causal graph: $Zerg[Marine.AttackPoint \rightarrow Health], Zerg[ID \rightarrow Health], Zerg[Marine.AttackTarget \rightarrow Health]$

The third solution is introducing addition relation classes for each kind of interaction. The fields of the class are the IDs of participants of the relation. For example, we may define a class $Attack$ for the attack relation, which has fields $Attack.Attacker, Attack.Target$. Then, we probably will have these causations in the causal graph: $Zerg[Attack.Target \rightarrow Health], Zerg[ID \rightarrow Health]$

Note that in the second and third solutions, the participants of the relations become a state variable in the environment and can be predicted by the model. Moreover, the network may be specially designed to better handle the $ID$ attributes with relational interactions.

About Weakness 3

Objects in OCR methods are usually homogeneous. Thus, OCR is compatible with our approach when the number of classes is exactly 1, while some adaptation of the network architecture and the loss function may be required. Unfortunately, the rebuttal period is too short to tune these details and perform an additional experiment. In the meantime, a solution limited to a single-class environment does not fully exploit the potential of an OOCDM.

The suggestion to adapt OCR to our work seems to be an interesting and valuable direction. In future work, we may try to extend OCR to multi-class scenarios (e.g., a one-hot vector in the feature of each object to identify its class, and then use different decoders for each class to obtain the attributes). However, much deeper work is needed to ensure that the representation follows causal symmetry.

Dear reviewer, we have submitted the revised version of our paper. In Section 4.1, we have added a sentence about the local causality to clear the confusion with respect to the article "Counterfactual Data Augmentation using Locally Factored Dynamics" (Pitis, et al), where they define "local causality" as the causal structure when $(\mathbf{S}, \mathbf{A})$ is confined in a local subset of the state-action space.

Thanks for the detailed feedback. Most of the concerns have been addressed and thanks for the clarification on the local causality concepts. I would keep my score for now and be attentive to the discussion with other reviewers.

Thank you very much for your questions and comments. We are happy to have a professional researcher in causal inference review our paper. We came up with this idea while trying to use causality to address other problems in RL last year, so it is possible that we have neglected some literature in the area of causal inference. Indeed, Relational Causal Discovery (RCD) has similar ideas to ours. Therefore, we will be sure to mention the related papers in the revised version.

About the question in Strengths

There are other attention-based models in object-oriented environments (usually considering homogeneous objects) using different architectures. In our work, the network is originally designed to compute according to the given causal structure.

About Weakness 1

This paragraph actually gives a summary of our work -- "generalizing causal dynamics learning to an object-oriented version" is our contribution, and "refining variables to consider only the relevant ones" is our motivation. I am not sure why these statements constitute the weaknesses of our work. The combination seems straightforward, but many issues need to be considered. This paper answers 1) under what condition is the same causal structure shared? 2) Exactly which variables are relevant? 2) what network architecture can be used for inference, and 4) how much does this reduce computational complexity. Meanwhile, we have demonstrated the effectiveness of this combination through experiments.

About weakness 2

I believe there is some necessity to use class-level causality in our setting. We focus on learning the causal structure in a Factored MDP because there have been a large number of studies using this structure to solve their problems, as mentioned in Section 2.1. The OOMDP is just a special case of the FMDP, so our model can directly serve all methods that work on the FMDP, as long as an object-oriented segmentation of the FMDP's variables is available. RCD involves another concept called "relations", which is not modeled in FMDPs or OOMDPs -- these relations may exist, but are implicit in the factorization. Consider the example of students taking courses (Fig. 2 in [1]), the relation "Take" may be implicitly described by two vectors $Alice.Take = (1, 0)$, $Bob.Take = (1, 1)$ in the FMDP. The OOMDP (FMDP) will only model these relationships as attributes (variables) in the state space, so class-level causality is needed to describe the causal graph when relations are not modeled or implicit (e.g., $Course[Student.Take \rightarrow Difficulty]$).

In addition, the classes and objects in OOMDP have already required a lot of domain knowledge about the environment. If the domain knowledge about relations is also required, the approach would be much more limited. Even a layman can instinctively know how to classify objects, but relations sometimes require experts. Meanwhile, RCD requires to learn a unique model for each type of relation, which is also very expensive, while our solution has a lower complexity.

About Questions

Scalability is embodied in the paper in many ways, not just in the sentence you quote. Of course, that sentence is already covered by the RCD papers, so I will add a reference there in the revised version. However, most of the claim about "scalability" is still original, embodied in the following aspects:

1. Part of the scalability comes from the Causal Symmetry (Eq. 4), which is not included in RCD. This symmetry allows us to use attention in the model because the input key and value vectors of attention are also symmetric. In this way, the model can capture the joint influence of other objects and manage different numbers of objects. In the RCD studies, their regression models do not have this property. As a result, in Fig. 9 of [1], the precision of causal discovery decreases when the number of entities increases from 2 to 3. However, the causal discovery of our model achieves higher precision with more entities.
2. RCD only concludes that the scalability of the causal discovery is improved, but we also examine the scalability of the prediction performance of the model. Many RL applications require the model to be accurate in predicting the next state. Table 3 in the main paper shows that OOCDM outperforms other causal models.
3. RCD does not conclude that scalability about computational cost. In some applications, such as model-based policy optimization, causal discovery is performed repeatedly in each iteration with new data collected. In large-scale environments, causal discovery can be excessively time-consuming, making the application of causal models intractable. In such applications, computational cost may be the most important aspect of scalability. In Table 2, we show that our object-oriented version of the model greatly reduces computation time.

[1] Marazopoulou et al. Learning the Structure of Causal Models with Relational and Temporal Dependence, UAI 2015

Dear reviewer, we have submitted the revised version of our paper. We have mentioned these papers in Related Works (Section 2):

1. Maier, et al. Learning causal models of relational domains, 2010.
2. Marazopoulou, et al. Learning the structure of causal models with relational and temporal dependence, 2015.