Identifiable Object Representations under Spatial Ambiguities

We thank the reviewer for their feedback and are glad that the reviewer found our experiments to be extensive and method to be robust.

The primary focus of the work is that it provides theoretical guarantees for identifiability in multi-view scenarios, and requires no viewpoint annotations, which builds upon the formalisms built in a single view scenario in Kori et al. (2024); Brady et al. (2023); Lachapelle et al. (2023). To the best of our knowledge, this is the first work addressing explicit formalisations of assumptions and theory required for achieving this. We also provide empirical evidence with synthetic datasets, where the transformation in distribution clearly demonstrates our claims.

In order to make the paper more self contained, based on other reviewers feedback we plan to include complexity argument and metrics details, along with model architectures as below.

VISA complexity: VISA achieves $\mathcal{O(VTNKd)}$ with the added complexity of $\mathcal{2O(VNd)}$ for inverse and forward view point transformation given by $\mathcal{T}_{\theta}$, while it retains the complexity per view to be $\mathcal{O(VTNKd)}$ which is the same as slot attention and probabilistic slot attention. Additionally, the representation matching function contributes $\mathcal{O(VK^3d)}$: this term does not alter the dominant term, in the general case when $K<<N$. Similar to PS, when VISA is combined with an additive decoder, the complexity of the decoder can be lowered due to the property of automatic relevance determination (ARD), eliminating the need to decode inactive slots.

SMCC details: we borrow the definition of SMCC from Kori et al., 2024. For two sets of slots $\\{\mathbf{s}_i\\} _{i=1}^M$ and $\\{\tilde{\mathbf{s}}_i\\} _{i=1}^M$, where $\mathbf{s}_i \in \mathbb{R}^{K \times d}, \tilde{\mathbf{s}} _i \in \mathbb{R}^{K \times d}$, extracted from $M$ scenes, the SMCC between any $\mathbf{s}$ and $\tilde{\mathbf{s}}$ is obtained by matching the slot representations and their order. The order is matched by mapping slots in $\tilde{\mathbf{s}}$ w.r.t $\mathbf{s}$ assigned by $\tau$,

followed by a learned affine mapping $\mathbf{A}$ between aligned $\tilde{\mathbf{s}}_{\tau(i)}$ and $\mathbf{s}$:

$\mathrm{SMCC}(\mathbf{s}, \tilde{\mathbf{s}}) \coloneqq \frac{1}{K\times d} \sum _{i=0}^K \sum _{j=0}^d \rho(\mathbf{s} _{ij}, \mathbf{A} \tilde{\mathbf{s}} _{\tau(i)j}).$

By design the SMCC metric is bounded between [0, 1], with the higher the value the better. We will add these details in the appendix.

Decoder architecture: As detailed in the paper we use two different classes decoder architectures (i) additive and (ii) non-additive, within additive we use both spatial broadcasting and MLP decoders, for non-additive we use transformer decoders. In terms of architectural, we follow SA(Locatello et al., 2020) for spatial broadcasting decoders and DINOSAUR(Seitzer et al., 2023) for both MLP and transformer decoder, we will describe architectures in detail in the appendix as in the response of reviewer zW6Z:

We thank the reviewer for their detailed feedback and are glad that the reviewer finds our paper well written, clear, and with reasonable claims and proofs. We are also glad that intuitions aided in the understanding of theorems and our claims.

As mentioned in the weakness section on page 8, the viewpoint sufficiency assumption is strong. It is questionable whether proposed model could be applied on real-world data

We do agree experiments on large real-world datasets would be helpful, however to the best of our knowledge there aren’t any real-world datasets with many multiple viewpoints: this is was the main motivation for proposing the synthetic dataset which consists of 72,000 x 5 number of images with 72,000 scenes with 5 viewpoints(in terms of variation it consists of 930 objects variety with 458 complex backgrounds).

Additionally, the primary focus of the work is that it provides theoretical guarantees for identifiability in multi-view scenarios, and requires no viewpoint annotations, which builds upon the formalisms in a single view scenario in Kori et al. (2024); Brady et al. (2023); Lachapelle et al. (2023). To the best of our knowledge, this is the first work addressing explicit formalisations of assumptions and theory required for achieving this.

Having said that, if the reviewer can point to any large real-world dataset we would be happy to consider it for the final version of the paper.

The evaluation metrics are not well-introduced. It is a little bit hard to understand how good the numbers represent.

Thanks for pointing this out. We borrow the definition of SMCC from Kori et al., 2024. For two sets of slots $\\{\mathbf{s}_i\\} _{i=1}^M$ and $\\{\tilde{\mathbf{s}}_i\\} _{i=1}^M$, where $\mathbf{s}_i \in \mathbb{R}^{K \times d}, \tilde{\mathbf{s}} _i \in \mathbb{R}^{K \times d}$, extracted from $M$ scenes, the SMCC between any $\mathbf{s}$ and $\tilde{\mathbf{s}}$ is obtained by matching the slot representations and their order. The order is matched by mapping slots in $\tilde{\mathbf{s}}$ w.r.t $\mathbf{s}$ assigned by $\tau$,

followed by a learned affine mapping $\mathbf{A}$ between aligned $\tilde{\mathbf{s}}_{\tau(i)}$ and $\mathbf{s}$:

$\mathrm{SMCC}(\mathbf{s}, \tilde{\mathbf{s}}) \coloneqq \frac{1}{K\times d} \sum _{i=0}^K \sum _{j=0}^d \rho(\mathbf{s} _{ij}, \mathbf{A} \tilde{\mathbf{s}} _{\tau(i)j}).$

By design the SMCC metric is bounded between [0, 1], with the higher the value the better. We will add these details in the appendix.

For example, could you elaborate more on 'vary by an affine transformation' in the caption of Figure 6? How do you evaluate and compare the results?

In the context of Figure 6 the behaviour of affine transformation implies the distribution across view sets, only differ in scale and translation. We do agree with complications in analysing higher dimensional latents. Figure 5-6 was our attempt on this visualising the feature-wise aggregated distribution to capture the trend. We have discussed the behaviour in the paper, but will expand it for clarity.

Additionally, we also have included a 2D-variant experiment in appendix G.1 where the distributions can be seen to be equivalent up to an affine transformation, validating our claims. Depending on space availability we will bring the 2D experiments to the main paper.

do you have some insights on how to ensure this assumption if other researchers are going to apply your model? The reviewer addresses this as a limitation, independent of the technical details within this assumption.

Theoretically, verifying the viewpoint sufficiency assumption is challenging. In some cases, as few as two viewpoints may be enough to satisfy this assumption, while in more complex scenes, adding additional views can improve visibility. However, beyond a certain number of viewpoints, we expect diminishing returns in performance, as the marginal gain from each additional view decreases. When we have control over the data generation process, we can adopt an adaptive viewpoint selection strategy. This could involve dynamically selecting views based on occlusion-aware heuristics or ensuring a larger set of viewpoints that are equidistant from the scene of interest while varying the angle and azimuth. This approach helps mitigate occlusion issues and ensures more robust object visibility, we will include this discussion in the main paper.

We thank the reviewer for their detailed feedback and are glad to see that the reviewer acknowledges our superior performance and believes the paper to address novel and important topic.

(W1)It is difficult to parse the curves in Figure 5 and 6. …

We do agree with complications in analysing higher dimensional latents: Figure 5 and 6 are an attempt to visualise the feature-wise aggregated distribution to capture the trend. While we have described the behaviour of this distribution in the discussion, we will expand it for clarity in the final version. Additionally, we also have included a 2D-variant experiment in appendix G.1 where the distributions can be seen to be equivalent up to an affine transformation validating our claims. We would also like to point to qualitative results included in appendix Figure 10-14. Depending on space availability we will bring the 2D experiments to the main paper.

(W2)It seems to be strongly connected to PSA while the writing of Section 4 …

As mentioned in a paragraph Viewpoint specific slots (L193-L214), the viewpoint specific slots are extracted with EM algorithm as proposed in Kori et al., 2024 (which is same as PSA) but applied on a transformed encoder output, with a transformation corresponding to a viewpoint specific inverse transformation. This is done to project all the slot representations for multiple views into a common vector space.

As pointed out by the reviewer and in the viewpoint specific slots paragraph, the algorithm here is the same as PSA with the only difference being with the transformation of inputs. We will cross reference section 3 in that paragraph to make this difference more explicit.

Additionally, the primary focus of the work is that it provides theoretical guarantees for identifiability in multi-view scenarios, and requires no viewpoint annotations, which builds upon the formalisms in a single view scenario in Kori et al. (2024); Brady et al. (2023); Lachapelle et al. (2023). To the best of our knowledge, this is the first work addressing explicit formalisations of assumptions and theory required for achieving this.

(W3) Meanwhile, the details of the model implementation are heavily missing…

Thanks for pointing this out, we will include them:

Decoder architecture: As detailed in the paper we use two different classes of decoder architectures: (i) additive and (ii) non-additive; within the additive architecture we use both spatial broadcasting and MLP decoders; for the non-additive architecture we use transformer decoders. Specifically, we follow SA (Locatello et al., 2020) for spatial broadcasting decoders and DINOSAUR (Seitzer et al., 2023) for both MLP and transformer decoder: for details about architectures please refer to the response of reviewer zW6Z

The K in equation 6 appears to be a hyperparameter of choice that …?

That's a valid point, however the dependency on K is inherent to SA and PSA, here we develop on these work to address spatial ambiguities by considering multiple viewpoints. Similar to the ARD study in Kori et al., 2024, during inference we did observe when K is set to higher that the required number, the model ignores the additional slots by assigning the mixing coefficient to 0, however lower K affects performance, similar to the ablations in Locatello et al., 2020.

If some object is completely occluded, how do you identify the object and its visibility without knowing the 3D prior of the environment?

That’s a great point. One of our key assumptions is viewpoint sufficiency, meaning that each object in the environment is visible in at least one of the considered viewpoints. If an object is completely occluded across all viewpoints, identifying it falls beyond the scope of this work. We will make this explicit in the paper

We thank the reviewer for their detailed feedback and are glad that the reviewer found our claims were supported by clear and concise evidence with correct proofs and sound experiments.

more extensive details on computational complexity, model training time, parameter counts

VISA complexity: VISA achieves $\mathcal{O(VTNKd)}$ with the added complexity of $\mathcal{2O(VNd)}$ for inverse and forward view point transformation given by $\mathcal{T}_{\theta}$, while it retains the complexity per view to be $\mathcal{O(VTNKd)}$ which is the same as slot attention and probabilistic slot attention. Additionally, the representation matching function contributes $\mathcal{O(VK^3d)}$: this term does not alter the dominant term, in the general case when $K<<N$. Similar to PS, when VISA is combined with an additive decoder, the complexity of the decoder can be lowered due to the property of automatic relevance determination (ARD), eliminating the need to decode inactive slots.

Decoder architecture: As detailed in the paper we use two different classes of decoder architectures: (i) additive and (ii) non-additive; within the additive architecture we use both spatial broadcasting and MLP decoders, for the non-additive architecture we use transformer decoders. Concretely, we follow SA(Locatello et al., 2020) for spatial broadcasting decoders and DINOSAUR(Seitzer et al., 2023) for both MLP and transformer decoder. In detail we use:

• spatial broadcasting decoders:

Input/Output: The generated slots are $\mathbf{s} \in \mathbb{R}^{K\times d}$, each slot representation is broadcasted onto a 2D grid of dimension $8 \times 8 \times d$ and augmented with position embeddings. Similar to slot attention, each such grid is decoded using a shared CNN to produce an output of size W × H × 4, where W and H are width and height of the image, respectively. The output channels encode RGB color channels and an (unnormalized) alpha mask. Further, we normalize the alpha masks with Softmax and perform convex combinations to obtain reconstruction.

Shared CNN architecture: 3 x [Conv (kernel = 5x5, stride=2), LeakyReLU(0.02)] + Conv (kernel = 3x3, stride=1), LeakyReLU(0.02) 

• MLP decoders:

Input/Output: similar to the spatial broadcasting decoder, here, each slot representation is broadcasted onto N tokens (resulting in $N \times d$) and augmented with position embeddings. Then the individual slot representation is transformed with a shared MLP decoder to generate a representation corresponding to feature dimension along with additional alpha mask, which is further normalised with Softmax and used in creating convex combinations to obtain reconstruction.

Shared MLP architecture: [Linear (d, d, bias = False), LayerNorm(d)] + 3 x [Linear (d, d_{hidden}), LeakyReLU(0.02)] +  Linear (d_{hidden}, d_{feature}+1) 

• Transformer decoders:

Input/Output: transformer consists of linear transformers encoder output $(N \times d_{feature})$ and extracted slots $(K \times d )$ as input, while returning the slot conditioned feature as output with a dimension of $(N \times d_{feature})$.

Transformer architecture: is made up of 4 transformer blocks, where each transformer block consists of a self-attention on input tokens, cross-attention with the set of slots, and residual two-layer MLP with hidden size $4 x d_{feature}$. Before the Transformer blocks, both the initial input and the slots are linearly transformed to $d_{feature}$, followed by a layer norm.

Model training time: As detailed in appendix G.7, our training usually takes between eight hours to a couple of days, depending on the model and the dataset. We run all our experiments on a cluster with a Nvidia NVIDIA L40 48GB GPU cards, we will add a pointer in the main text.

extensive ablations on larger real-world datasets would strengthen scalability claims.

We do agree experiments on large real-world datasets would be helpful, however to the best of our knowledge, there aren’t any real-world datasets with many multiple viewpoints: this was the main motivation for proposing the synthetic dataset which consists of 72,000 x 5 number of images with 72,000 scenes with 5 viewpoints; in terms of variation it consists of 930 objects variety with 458 complex backgrounds.

Additionally, the primary focus of the work is to provide theoretical guarantees for identifiability in multi-view scenarios, while requiring no viewpoint annotations, which builds upon the formalisms in a single view scenario in Kori et al. (2024); Brady et al. (2023); Lachapelle et al. (2023). To the best of our knowledge, this is the first work addressing explicit formalisations of assumptions and theory required for achieving this.

Having said that, if the reviewer can point to any large real-world dataset we would be happy to consider it for the final version of the paper.