A compressive-expressive communication framework for compositional representations

Rebuttal Reviewer ct41

We thank the reviewer for their deep and insightful response and valuable suggestions, which we believe will significantly improve the paper.

---

Weaknesses / Suggestions

1. We appreciate the suggestion to expand the introduction and related work to provide readers with necessary background on the model components and pressures in the iterated learning setup, which our strategy challenges. We agree this will help clarify the paper's positioning—thank you again for the recommendation!

2. This suggestion has two parts:

   1. We agree that the link to chain-of-thought (CoT) is speculative and partial. The only shared property is that CoT reasoning steps, like our message tokens, exhibit compositional and sequential structure. However, CoT subsequences are not explicitly pressured to lead to a correct conclusion/output, making the analogy misleading. We plan to remove this comparison.
   2. Thank you for suggesting we clarify the connection between progressive decoding and compositional metrics like topographic similarity. While we mention this in line 127, we will add a clarifying sentence in the final version.

3. Thank you for highlighting the connection to object-centric learning, slot-type architectures, and additive decoders. We will add a discussion of these in the related work section.

4. We appreciate the suggestion to broaden the scope beyond emergent communication. We are indeed exploring further uses of our signaling architectures for representation learning, and the idea of cross-model communication is promising if fine-tuned appropriately. We will add a discussion of this in the "Limitations and Future Work" section.

5. We agree with the suggestion to move Proposition 3.1 to the appendix and will do so in the final version.

6. Regarding Table 1:
   To assess statistical significance, we added 5 seeds per method and ran permutation tests, confirming that the impact of our components on Topsim is statistically significant. The updated table and p-values are in the rebuttal to reviewer Likm; results for Table 2 are in the rebuttal to reviewer rCnw.
   Concerning inductive biases, we note that gains in Topsim are stronger when combining PD+FiSI+PDM, compared to applying them separately. That the components are less effective individually does not contradict our claims. On the contrary, it supports the motivation for combining them—each pressure alone may be insufficient, but together they produce synergy.
   For example, FiSI is meant to allow sender strategy drift, but this drift doesn't arise naturally from optimization. Hence, we introduce PDM to explicitly encourage such exploration. We will clarify this in the final version.

---

Questions

1. Progressive reconstruction loss:

   1. This loss clearly enforces (1)—penalizing longer messages—since later tokens contribute more to the loss. As for (2)—penalizing tokens that carry no information—we do not claim this. Instead, we argue it promotes interpretability in submessages (prefixes of increasing length), as the receiver is penalized for failure to reconstruct from partial messages.
   2. Encouraging shorter messages aligns with a core pressure in Iterated Learning theory. Thus, we prefer regularizers that enforce both compression and interpretability (e.g., progressive loss) over those that only target informativeness. Efficient encoding is a necessary (though not sufficient) condition for low Kolmogorov complexity, which underlies emergent compositionality. Appendix F explains why our regularization is needed, as training data covers only a small portion of possible factor combinations.

2. Use with continuous features:
   In principle, our setup could be extended to continuous features. However, in practice, discrete and separated encodings have proven more effective for compositionality in iterated learning (see [45]). Discreteness mirrors human vocabulary and enables concept separation.
   Technically, discreteness supports a richer encoding space and helps avoid mode collapse, as shown in VQ-VAE literature. We believe this is especially important in combinatorially extended message spaces like ours, where collapse risks are amplified. We will emphasize this point in the related work section, linking it to object-centric learning.

3. Clarification on λ and TopSim:
   Thank you for catching this. We previously wrote (lines 238–239) that “larger values of λ encourage more informative messages, but reduce alignment with generating factors,” to justify the claim that “intermediate values of λ are best.” As you noted, this isn't clearly supported by the data. What does drive the optimal λ value is the emergence of a plateau in TopSim, while useful length continues to increase. We will correct this explanation.

4. Clarification on “collapse”:
   In hindsight, our use of “collapse” was poorly worded. We appreciate the reviewer pointing this out. What we meant was that, without additional pressure, a new sender has no incentive to drift from the previous sender’s trajectory—i.e., the decoding sequence from submessages of increasing length remains unchanged.
   By adding entropy regularization, we encourage exploration of alternative encoding strategies, similar to how entropy is used to promote exploration in reinforcement learning. The actual “collapse” occurs over generations. We will revise the wording accordingly.

5. Trade-off between expressivity and compressibility:
   By this, we mean that adding PDM improves TopSim (from 0.141 to 0.16), but the other two metrics worsen (see second vs. last row in the second part of Table 1). This contrasts with the reviewer’s statement that PDM “has not further increased the structure or comprehensibility of the language (TopSim),” so we may be misunderstanding their comment. Did we miss something?

Rebuttal Reviewer gfpt

We thank the reviewer for their time and thoughtful response.

The reviewer's concerns primarily highlight the need for evaluation on more realistic or complex datasets, or the use of English for communication. This may reflect a misunderstanding of our paper’s main positioning, which focuses on emergent language via iterated learning. In this setting, controlled scenarios are useful in multiple ways, and using English as a medium of emergent communication is not straightforward. We expand on this below.

---

Weaknesses

1. Evaluation via non-synthetic problems:
   Thank you for raising this important point.

   We acknowledge that our experiments are conducted on synthetic datasets. This is intentional and aligned with common practice in the emergent communication and compositional generalization literature. Controlled environments are essential for systematically studying inductive biases and the emergence of structured communication protocols (indeed, nearly all cited references on emergent language use synthetic datasets of comparable complexity—e.g., [5], [12], [14], [23], [24], [35], [37], [38], [44], [47], [48], [52], [58], [64]).

   Specifically, synthetic datasets provide several benefits:

   1. Controlled data distributions: They allow precise control over training and test distributions—especially important for implementing compositional splits that test generalization to novel attribute combinations, which is central to our study.
   2. Ground-truth factorization: Meaningful evaluation of compositionality requires access to generative factors (e.g., shape, color, size). Real-world datasets often lack this disentangled ground truth, making metrics like topographic similarity difficult or ambiguous to apply.
   3. Complexity minimization: Following Occam’s razor, we avoid unnecessary complexity to preserve clarity in causal interpretation. Introducing relational reasoning or complex object structures would increase dimensionality and introduce new, confounding effects that could obscure the impact of our proposed mechanisms.

   While evaluating more complex and realistic tasks is a valuable future direction (as noted below), such extensions introduce orthogonal challenges. We believe our current setup provides a principled and interpretable testbed for isolating the effects of our mechanisms on compositionality. We’ve added a note about this extension path in the discussion section.

2. English as a communication medium:
   We appreciate the reviewer’s thoughtful comment. However, this concern seems to arise from a misunderstanding of our objective.

   Our framework does not aim to model or analyze evolved human languages such as English, but instead studies the emergence of compositional communication protocols de novo between artificial agents (see lines 44–47). This is motivated by foundational questions in cognitive science [29], where a key challenge is understanding how compositional structures arise from basic communicative dynamics and learning mechanisms.

   A core theoretical motivation in studying emergent language is to test whether compositionality emerges naturally from optimizing for efficient and generalizable communication, as proposed by recent information-theoretic accounts [43]. Thus, our work should be viewed in the context of emergent communication and compositional learning, not natural language modeling.

   While English is indeed compositional, it is the product of centuries of cultural evolution shaped by complex social and historical factors—many of which are not tractable in computational models. Emergent languages, in contrast, enable precise control and isolation of contributing factors, allowing systematic analysis of their role in shaping communication.

3. Multi-object scenes and relational reasoning:

   1. We respectfully disagree with the claim that our work does not evaluate compositional generalization. As noted in line 212, we adopt the compositional generalization split introduced by [52], which is specifically designed to test generalization to novel combinations of known factors—a core benchmark for compositionality. We apply this to two well-established datasets, Shapes3D and MPI3D, widely used for analyzing compositional structure in learned representations.
   2. Evaluating multi-object scenes with relational queries is indeed an exciting direction, but we believe it is outside the scope of our current study. Our goal is to isolate and understand how our mechanisms affect emergent message structure and efficiency. Adding relational reasoning would introduce additional complexity and confounds, making attribution to our contributions less clear.
   3. We focus intentionally on the simplest form of compositionality: generalization over novel combinations of independent latent factors (e.g., shape, color, size). We do not address more complex compositional structures (e.g., syntax or multi-object relational reasoning), which introduce additional challenges (see 1.iii above).

   Nonetheless, we appreciate the reviewer’s suggestion and plan to explore relational settings in future work.

---

References

[29] Simon Kirby et al., “Compression and communication in the cultural evolution of linguistic structure,” Cognition, 2015.
[43] Yi Ren et al., “Compositional languages emerge in a neural iterated learning model,” 2020.
[52] Lukas Schott et al., “Visual representation learning does not generalize strongly within the same domain,” ICLR, 2022.

---

Questions

To our understanding, the reviewer asks how Eq. (3) contrasts with Eq. (1) used for progressive decoding (PD). The key distinction is that Eq. (3) applies to the imitation phase, whereas Eq. (1) pertains to the interaction phase, which has significant implications:

1. In the interaction phase (Eq. 1), both sender and receiver are unfrozen, and the loss measures reconstruction accuracy from input to output. In the imitation phase (Eq. 3), the receiver is frozen, and the sender imitates the reconstruction produced by the previous-generation sender, not the ground truth (see line 172).
2. The left and right components of Eq. (3) compare full vs. final reconstruction. Our choice—to use only final state reconstruction during imitation (as in our Eq. 2, corresponding to the right-hand side of Eq. 3)—supports broader sender exploration when combined with PDM (see §3.2.2). This differs fundamentally from PD, where sender and receiver jointly optimize under an efficiency constraint (minimizing useful message length) imposed by Eq. (1).

We hope this clears up the confusion—please let us know if we can clarify further.

Dear Reviewer,

Thank you very much for taking the time to review this paper. Would you kindly confirm whether the authors' response adequately addresses the third concern you raised under "Weaknesses"?

Best regards, AC

Rebuttal Reviewer rCnw

We thank the reviewer for their time and thoughtful response.

The reviewer asks whether our paper is primarily about emergent communication or compositional generalization. In hindsight, we recognize the abstract may have caused confusion. Our main focus is emergent communication, and we will clarify this in the revised abstract.

Weaknesses

1. Response to two concerns: (reference numbers [1], [2], [3] as in the reviewer's text, [64] as in our paper's draft)

   1. Table 1 and lines 331, 248–250: The reviewer rightly notes the original Table 1 showed marginal improvements. We reran all methods with 5 additional seeds to increase statistical power and performed permutation tests across interaction, imitation, and regularization methods (see response to reviewer Likm). We’ll include these in the supplementary materials. The updated table shows a clearer trend—final state imitation generally outperforms message imitation—but statistical significance was only observed in MPI3D. We will clarify this.
   2. Positioning beyond emergent communication: The main insight is that pressures for signal complexity (useful length), expressivity (reconstruction error), and compositionality (topsim) arise from different aspects of the interaction setup and must be balanced for more compositional representations. This underlies our three innovations. While generalizing beyond emergent communication is a broader debate (e.g., [1]), our approach resembles complexity regularization methods like self-distillation [2]. We will elaborate on this in the "Limitations and Future Work" section.

2. Progressive Decoding nontriviality: PD acts as a length penalty, but the trade-off it induces with reconstruction accuracy is nuanced. As shown in Figure 3 (App. D), PD can prioritize efficiency (shorter useful length) over accuracy. The tuning of λ becomes critical. Partial decoding does not necessarily improve reconstruction but enhances efficiency. The key challenge is balancing concise and expressive messages while avoiding ambiguity.

3. Table 2 – two clarifications:

   1. RMSE metric: Table 2’s RMSE does not reflect reconstruction error but evaluates representations on a downstream regression task, following [64]. This differs from true reconstruction performance and is less directly tied to compositionality and interpretability (similar to discriminative metrics in 4.iii). We'll clarify this distinction and its implications in the final version, as also noted by reviewer Likm.
   2. Accuracy and significance: We increased each method to 10 runs (see updated table) and ran permutation tests comparing CELEBI to the EL baseline. Relevant comparisons: In Shapes3D, CELEBI outperforms EL in accuracy (p=0.0017), RMSE (p=0.001), and disentanglement (p=0.0041). In MPI3D, CELEBI exceeds EL in RMSE (p=0.0009) and disentanglement (p=0.0001). Full results will be in the paper.

4. Multiple points:

   1. Interpretability metrics: Following the suggestion, we added synonymy, homonymy, freedom, and disentanglement (equal to "1 - entanglement" from Conklin et al. [1]) to the supplementary. Permutation tests show PD and PDM reduce entanglement. In Shapes3D, PD outperforms the interaction baseline (p=0.00059, diff=0.041); in MPI3D (p=0.0425, diff=0.064). PDM beats KoLeo (p=0.0052, diff=0.0195). Since reducing entanglement requires consistent encoding of different meanings, this supports PD's role in producing more compositional representations. We’ll expand on this in the final version.
   2. Qualitative results: Besides the above metrics which we will add in a table, Appendix I contains generated examples showing regularities. We’ll add comparisons across methods to highlight improvements.
   3. Representation alignment from [3]: In contrast to [3], where representations were not explicitly aligned with visual features, our setup uses a reconstruction task. Since agents must reconstruct images, the emergent language aligns naturally with underlying visual features. Topsim emerges from the correlation between generative factors and pixels—not class labels—ensuring grounded representations.
   4. Message structure: Intermediate reconstructions often show meaningful transitions (App. I), unlike denoising noise. We'll add more examples to strengthen this point.

5. Comparison with LazImpa: Our PD loss is conceptually simpler and more interpretable. LazImpa's penalties depend on model-specific accuracy predictions, whereas ours apply a single global term (eq. 1), creating a clearer, system-wide efficiency pressure. Unlike LazImpa, where message length pressure is split across roles, PD treats the system as a whole. We’ll clarify this conceptual distinction in the revised version.

---

Minor Issues

1. We will add takeaways in figure captions.
2. We will merge overlapping background sections.
3. We’ll revise to: "...favoring easy to learn languages..."
4. Thank you!
5. We will use "experimental setup" instead of "methodology."
6. The word “increase” refers to worse reconstruction, which we believe is correct.

---

Questions

1. We have run permutation tests (see reviewer Likm) and will include them in the supplementary material.
2. Topsim is computed using generating factors. We will clarify this. We’ll also add qualitative examples showing the effect of changing message symbols on reconstruction. Without PDF support, we’ll reference these in Appendix I and describe the new figures.
3. Effect of λ: As λ increases, the PD loss asymptotically resembles full reconstruction, since the highest-order term dominates. In practice, λ ≈ 2.5 already saturates useful length. We’ll extend Figure 2 to λ=10 and explain this in §3.1.

---

Table: Disentanglement Metrics for Shapes3D and MPI3D

Rebuttal Reviewer Likm

We thank the reviewer for their time and their attentive response.

We believe that a misunderstanding regarding the positioning of our paper underlies several of the reviewer’s concerns. Our focus is on proposing a new iterated learning framework in which emergent representations become compositional. We do not aim to resolve the ongoing debate around the relationship or trade-off between compositionality/disentanglement and downstream accuracy. This is especially relevant for the interpretation of Table 2 (see discussion below). The three main new components of our framew...

---

Weaknesses

1. Regarding Table 1:
   While the reviewer did not mention this table explicitly, we believe their remarks about “only slight improvement” may refer to it. Several reviewers pointed out that results in Table 1 may not be statistically significant.
   To address this:

   • We increased the number of runs per method to 10 (updated tables below).
   • We conducted permutation tests across methods in interaction, imitation, and regularization.
     Results indicate statistically significant differences:
   • Shapes3D: Progressive decoding outperforms the baseline in Topsim (p = 0.0233) and useful length (p = 0.0031). PDM outperforms KoLeo in Topsim (p = 0.0004) and reconstruction error (p = 0.0431).
   • MPI3D: FiSI outperforms message imitation in Topsim (p = 0.0007) and reconstruction error (p = 0.0201), and full state imitation in Topsim (p = 0.0019). PD outperforms full message reconstruction in useful length (p = 0.0201).
     Full results will be included in the final paper. Let us know if further comparisons are needed.

2. Regarding Table 2:
   As noted by the reviewer, our method improves disentanglement but underperforms in accuracy relative to continuous VAEs, while slightly outperforming the discrete baseline. These outcomes align with literature (see [1]): disentanglement often harms accuracy in downstream classification.
   Discrete representations may be less accessible to simple classifiers (as shown in [1]), and inductive biases for compositionality do not imply disentanglement [2].
   Thus, the disentanglement results in Table 2—not accuracy—are central to our paper’s positioning. We acknowledge this was not clearly conveyed and will revise the discussion accordingly.

3. On the nature and interpretability of emergent representations:
   Thank you for raising this. Topsim measures structural alignment between representation and semantic space, but not characteristics of the emergent language itself.
   Following reviewer rCnw’s suggestion, we added metrics like Synonymy and Homonymy, and discussed them in the supplement (see also Weakness 4.i in their rebuttal). Our focus is on basic compositional properties; deeper interpretability metrics are outside the paper’s scope but worth future exploration.
   Appendix I already shows how initial submessages carry relevant information—captured by our useful length metric. We will also provide more qualitative examples to support this.

---

Questions

1. Thank you for pointing this out. We agree the original equation was overly condensed and have revised the notation for clarity.

2. Yes, $C$ is fixed and large enough that possible messages far outnumber dataset states. This ensures meaningful compression in useful length (see line 221) without harming reconstruction accuracy. We will clarify this in the text.

3. In line 134, the sentence “$\lambda > 1$ achieves the best performance” is a typo—should be “$\lambda \approx 1.5$.” Eq. (1) allows $\lambda > 0$, not just $\geq 1$ as the reviewer inferred (see Figure 2).
   Also, $C$ is fixed in Eq. (1). As $\lambda$ increases, more weight is placed on later tokens. In the limit $\lambda o \infty$, PD becomes equivalent to full message reconstruction. Even when $\lambda = 1$, earlier tokens still contribute to more terms than later ones and are thus more heavily pressured. We will clarify this.

Once again, we thank the reviewer for their time and valuable feedback.

---

Shapes3D (10 runs)

---

MPI3D (10 runs)

---

[1] Zhenlin Xu, Marc Niethammer, and Colin Raffel. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language, 2022. arXiv:2210.00482
[2] Milton Llera Montero et al. The Role of Disentanglement in Generalisation, ICLR 2021. OpenReview