We are grateful for your valuable and supportive feedback. Below, we provide clarifications and address your questions.

Model diversity:

While the main text indeed presents experiments using GPT‑J, the supplementary material already includes the cross‑evaluation experiment also with Llama 3.1 8B and GPT‑NeoX 8B, where we observe essentially the same results as for GPT-J. For the camera-ready version:

• We will extend this analysis and include all of the remaining experiments on Llama 3.1 8B and GPT‑NeoX 8B.
• We will also explicitly reference these additional results in the main text to ensure readers can easily locate and interpret them.
• We will also include a figure comparing performance across the different language models.

Language diversity:

While our method is language-agnostic in principle, evaluating its performance across different languages would require curated cross-lingual relational datasets, which are beyond the scope of this paper. Nonetheless, we recognize that a thorough multilingual analysis is an important and promising direction for future research.

TT-SVD and order-4 tensor networks:

Our experimental setup is specific to order-3 tensors (it requires a subject, relation and object embeddings on the free legs). We considered adding additional contextual information via a fourth leg, but this extension is beyond the scope of the present paper.

Thank you for suggesting TT-SVD. We hypothesize that the supervised signal provided by the cross-entropy loss in end-to-end training plays a crucial role in guiding what information is retained and compressed within the tensor. Since our training method already proved effective in practice, we did not pursue alternative training or tensor factorization approaches.

LoRA:

Thank you for bringing up the idea of applying our framework in the context of LoRA. We agree this is a compelling and promising direction, for several reasons:

• A single low-rank LoRA matrix can be expressed with an order-2 tensor network with basically an arbitrary internal structure.

• A setup which is more analogous to our paper is to consider LoRA matrices collectively. We might collect the low‑rank matrices produced by a single LoRA run across an entire network. Alternatively, we could take the matrices obtained from several independent LoRA trainings. In either case, a tensor‑network framework can act as a unified, highly compressed representation of all the LoRA matrices. Such setups can also take contextual information into account on a leg of a tensor network.

• More broadly, we find this line of thinking highly relevant to mixture-of-experts models, where structured compression and modular representation are increasingly important.

We appreciate the suggestion, we will mention this in the paper. We believe it opens up several interesting avenues for future work.

We thank you again for the feedback and we hope the clarifications above address all remaining concerns.

We sincerely appreciate your thoughtful and constructive feedback. We address your questions in detail and provide the requested clarifications below.

Training details of Figure 1a:

Throughout our paper we always use the ground-truth values for the target outputs.

In Figure 1a, we evaluated faithfulness on the set of training (subject, object) pairs. Our intention was to use compressibility as a tool to understand and interpret the decoders’ structural and semantic similarities.

New results on a (subject, object)-wise train/test split

To avoid any confusions however, during the rebuttal period we conducted experiments where we made a train/test split on the (subject, object) pairs.

To ensure a comprehensive evaluation, we performed experiments on the original dataset as well as the extended and mathematical variants. These results show that on the mathematical dataset the faithfulness value on the test set is close to perfect, 0.97. On the original and extended datasets, we observe weaker generalization: the model exceeds the majority-guess baseline in only about half of the relations and falls short in the remainder, yielding an overall test-faithfulness mean of 0.31. We attribute this drop in performance to having too little training data for our end-to-end training setup (which is naturally expected to be prone to memorization-related issues). This is not the case in the mathematical dataset however: sufficient amount of samples enable this strong generalization (both sample-wise and relation-wise).

In the camera-ready version of our paper we will elaborate on this level of generalization as well, and include a new figure and analysis with the above content.

Baselines

In Figure 1a, we use the entire set of (subject-object) pairs for a relation. By contrast, the linear decoder baseline follows the Jacobian‑based procedure of Hernandez et al., which estimates each LRE from a couple of examples (in particular, we used 8 examples). We selected this baseline to provide a more canonical reference point. We side with the reviewer that our evaluation could benefit from other baselines as well. We outline several options below that we will include in the camera ready.

Additional baseline: separately trained LREs

Two natural options are:

• Full-rank LREs: we train the LREs for each given relation separately.
• Low‑rank LRE variants: we train the LREs for each given relation separately and use a low-rank parametrization.

During the rebuttal period we implemented both of these additional baselines to provide a clearer picture. We trained both of these new baselines in the similar end-to-end manner that we used throughout the paper, and also implemented the (subject, object) train/test splitting method discussed above.

Tensor networks outperform the “naive approach”

Unsurprisingly, the results show that indeed separately trained LREs attain high faithfulness—we have an enormous amount of parameters.

Not just more importantly but perhaps even more surprisingly, even with these strong baselines in place, the tensor‑network achieves a better faithfulness‑per‑parameter trade‑off. In particular, the tensor network delivers higher faithfulness than low-rank LREs with even less parameter count.

For example:

Additional baseline: randomize relation embeddings

During the rebuttal period, we have implemented the suggested baseline of randomized relation embeddings and tested it on the mathematical dataset. These results indicate that the absence of the semantic information in relation embeddings results in a drastic performance drop on held-out (subject, object) pairs, whereas training samples can be memorized perfectly.

We would like to note that the generalization experiments (Figure 4) already demonstrate that the tensor network do make use of the relation embeddings, otherwise, it could not succeed in predicting the object for subjects that correspond to more than one relations in the math dataset (e.g., 13+6=19, 13-3=10). The fact that it exploits even the semantic structure (and do not simply treat relation embeddings as simple discriminative labels) is demonstrated by the positive generalization results on the relation-wise held-out test set.

Additional baseline: randomize subject/object embeddings

We also conducted the suggested experiment with randomized subjects and object embeddings.

Specifically, we mapped each distinct subject and object to a random vector and used this embedding throughout the training. In our experiments, the tensor network failed to memorize the training set and to generalize on the held-out pairs as well (resulting in a faithfulness of 0).

Changes to the manuscript

We plan to implement the following changes to the manuscript:

• We will include the sample-wise train/test split results, with a new figure and brief analysis.
• We will add clarifications regarding the baselines (as discussed above) in a new subsection of Section 3 and in the technical details of Section 2.
• We will update Figure 1a to reflect the full-rank and low-rank LRE baselines.
• We will discuss the training with randomized relation and (subject, object) embeddings in a new subsection of Section 3.
• Thank you for caching the rank/order terminology issue. Will adopt your suggestion in the revised version.

We thank you again for the constructive feedback, and we hope we have addressed all your concerns. We remain available to discuss any further questions during the discussion period.

We thank you for the positive and valuable feedback; below we address each question and detail the updates in the paper we will make based on them.

• We ran most of our experiments on GPT-J simply because it is the model we started to work with.

• Our cross-evaluation experiments with Llama 3.1 8B and GPT-NeoX 8B exhibit strikingly similar behavior to GPT-J, leading us to anticipate similar outcomes across other experiments as well. In the camera-ready version, we will include these figures also in the appendix, and will refer to these additional results more explicitly in the main text of the paper.

• Regarding the legibility of our figures, we will increase the font sizes and refine labels—we can take advantage of the extra page in the camera‑ready version to do so.

• We will also reformat the left quotes.

• We appreciate the suggestion to train smaller LMs from scratch to eliminate any potential pre‑training exposure to the test data. While this is indeed valuable, the scope of our paper is to decode and compactly represent the knowledge already present in pretrained LMs. Exploring this question would address a promising, but rather different research direction: how relational knowledge is acquired under constrained training.

• Wall-clock time: the models train for about 3 hours on a single H100 GPU. Much of the reported 5000 GPU hours were spent on grid searches and frequent evaluations conducted to support a detailed analysis of model behavior.

• GPU memory: for example, in the case of the simple tensor network, the total parameter count is $d \cdot d_s + d \cdot d_o + d \cdot d_r + d_s \cdot d_r \cdot d_o$, where $d$ is the embedding dimension of the LLM, and $d_r, d_s, d_o$ are the inner dimensions order-3 tensor in the core. Notably, when substantial compression was achieved, the core tensor parameter count $d_s \cdot d_r \cdot d_o$ was significantly lower than that of the projection matrices ($d \cdot d_s + d \cdot d_o + d \cdot d_r$). We will include a brief and exact analysis of the parameter counts and memory usage in the main text, as it would indeed enhance the quality and clarity of the paper.

We thank you again for the constructive feedback, and we hope we have addressed all your concerns. We remain available to discuss any further questions during the discussion period.

Thank you for your valuable suggestions. We will apply the following changes to further improve our paper:

• We will include the missing citations.

• To improve readability in 2b and 3b, we will enlarge fonts and refine labels.

• We will move Figure 4a to the Appendix and only cover its content in the main text. In its place, we will add additional visualizations—e.g., a figure comparing the results of the three evaluated models (GPT‑J, Llama 3.1 8B, and GPT‑NeoX 8B).

• Regarding 2a and 3a, we explored multiple alternatives before settling on the current versions. Our view is that these convey crucial global context: they show that the cross‑evaluation matrices are not diagonal—the surprising observation that motivated our investigation into the semantic structure of relations. Without illustrating this bigger picture, we believe we would lose important context for the reader. With the additional page in the camera‑ready version, we believe that there will be sufficient space to keep this figure in the main text.

• Thank you for drawing our attention to Chanin et al. (2024), as you pointed out, they explore a related but different idea. Their work focuses on (relation, object) pairs with LRCs represented ultimately as vectors, whereas our work focuses on the relations, and the LRE functions themselves. For this reason, we would not interpret LRCs as property extractors in our framework, but there is indeed a meaningful interplay between the notions and approaches presented in the two papers. We will add the reference and a brief paragraph to discuss the relation between the two.

We are again grateful for your positive feedback. We hope that our responses fully address your suggestions. We are happy to elaborate on any remaining issues throughout the discussion period.