ARC-AGI Without Pretraining

We thank the Reviewer for their thoughtful evaluation of our work, and their comments which help us to improve our work and it's presentation.

Addressing weakness #1: See below for rebuttal to question 2.

On weakness #2:

Multiple reviewers have pointed this out, and we agree with this assessment. To improve the clarity of our method as written, we will choose to include three algorithm/pseudocode blocks in Section 3 in the camera ready version. See our rebuttal to Reviewer DdUE for these algorithms. Algorithms 2 and 3 will contain the relevant loss functions. For reproducibility, the code for reproducing the results is already in the supplimentary material. This contains full instructions for usage as well as tips to help you read the code.

Otherwise, we completely agree with all the rest of the strengths and weaknesses listed.

For the questions:

1. We will also choose to add a table with baselines/comparisons to Section 5. This table is shown in our rebuttal to Reviewer rsDF. It will include the method "Test-Time Training" [1], which is pre-trained and also trains on the test-time puzzle. We do not know of other results that train exclusively on the test-time puzzle, if that is what you are looking for. Due to timing constraints of the rebuttal period, we are unable to run/show any experiments to compare against direct supervised learning during test time.

2. This work is mostly a proof of concept for the MDL approach towards intelligence. Most of the methodology is not transferrable to other downstream tasks. The main part that is transferrable, is the idea of how to transform the MDL objective into a problem of gradient descent during inference time, as explained in Section 3. This is because the statements of Section 3 can still apply when we replace:

   • "solving ARC-AGI puzzles" -> "infilling partially observed data"
   • "incomplete puzzle" -> "observed data"
   • "answers" -> "unobserved data"

   Looking at our method as a data infilling methodology may help other researchers make similar derivations of more new methods for other domains like supervised learning and generative modeling.

   While we cannot add any empirical results on any such method (our architecture is ARC-AGI-specific,) we can posit this extension as a theoretical possibility in the camera-ready version of the paper if the Reviewer would like. That said, we feel that this kind of suggestion may be too speculative and unsubstantiated to include in published work, and we would otherwise like to avoid it.

3. See our address to weakness #2.

4. We are unsure which other reconstruction objectives the Reviewer wants us to discuss/experiment with. Two possibilities that we can cover:

   • Other unsupervised learning approaches known to the literature. Continuing discussion from the Reviewer's question 1, the new baselines/comparisons table will also include the method "Latent Program Network" [2]. Latent Program Network includes a VAE setup, which is an unsupervised learning approach with a reconstruction objective. This method is also mentioned in the second bullet point in Appendix H, but we will now also mention it's unsupervised objective in the table as well.
   • The Reviewer wants us to use a reconstruction objective ourselves. CompressARC does indeed use a reconstruction objective, see crossentropy in Algorithm 2 from our rebuttal to Reviewer DdUE. The loss we compute in Algorithm 3 has two components, one of which is the reconstruction objective. We apologize for any confusion caused by our original description of this.

5. We are unable to get this experiment set up in time, but we agree this is a good idea, and will report variance in the next version of this paper. To try to alleviate the Reviewer's concerns, we remind the Reviewer by introducing Algorithm 3 that an entirely separate architecture is trained for every puzzle, and a different quantity of parameters are initialized and learned in each case. Since the initialization must have changed for every puzzle, there is already a degree of per-puzzle random variation involved that reduces the ways in which bias can enter our experiments.

We thank the Reviewer once again for the discussion, and we hope we have addressed their questions satisfactorily. We invite the Reviewer to discuss further on any aspect of our work, if they so desire.

References:

[1] Akyürek, Ekin, et al. "The surprising effectiveness of test-time training for few-shot learning." Forty-second International Conference on Machine Learning, 2024.

[2] Macfarlane, Matthew V., and Clément Bonnet. "Searching latent program spaces." arXiv preprint arXiv:2411.08706 (2024).

We thank the Reviewer for their thoughtful review and their suggestions on how to improve the clarity of our paper.

On strengths: We thank the Reviewer for recognizing the generalization ability of our highly data-limited method.

On presentation and clarity:

• Thank you for the suggestion, we agree that an algorithm section would improve the clarity. We will replace lines 110-123 with a new section, see Algorithm 2 in our rebuttal for Reviewer DdUE. For consistency, we will also change lines 126-129 to an algorithm section as well. (See Algorithm 1 in our rebuttal for Reviewer DdUE.)

• Seeing that the architecture is the part of the paper that is the least insightful/transferrable to other domains, we decided to only give a brief overview in the main text, and include a fully detailed description of the architecture's design in Appendices B and C for more enthusiastic readers and for completeness.

• We would like to emphasize in Section 4 that the architecture is heavily engineered to be used specifically for ARC-AGI, and that the engineering details are less transferrable to other domains, making them less important to include than, say, the broader idea of making the architecture respect lots of symmetries. The reader learns less from these details, so they are instead included in Appendices B and C for completeness and for more eager readers. If we would like to keep Section 4 brief for this reason, may we offer to the Reviewer to replace the sentence on line 149-150 with the following?

  We warn the reader that the model components are heavily engineered and very domain-specific. This makes them less informative on the overall idea behind CompressARC's approach, which merely requires that the architecture as a whole respects the symmetries of the problem domain. As such, we merely list the architecture components here, and describe each one in full detail in the Appendix:

  We agree that Section 2 does not need its current amount of detail. In the camera ready version, we will move most of it's detail to Appendix K, to make space for the new algorithm sections.

On the unclear learning objective:

We apologize for the confusion here. We hope the suggested algorithm sections clear this up. Algorithm 1 serves as the literal description in MDL, and Algorithm 2 minimizes its length. Algorithm 2 produces the puzzle's solution as an intermediate value, so Algorithm 3, which is CompressARC (see Algorithm 3 in our rebuttal for Reviewer DdUE,) is just a truncated version of Algorithm 2 that only outputs the solution. The reconstruction term in the loss calculates the expected length of line 3 of Algorithm 1, and the KL term in the loss calculates the expected length of line 1. This is where the connection between the loss function and the MDL comes from.

On the missing baselines:

Thank you for the suggestion, we agree that adding baselines will better contextualize and strengthen the claims in our paper. We will include the following information in a new table in Section 5:

These numbers obtained directly from the cited papers. We will move Table 1 to the Appendix to make space for this new table.

We thank the Reviewer once again for the discussion, and we hope we have addressed their questions satisfactorily. We invite the Reviewer to discuss further on any aspect of our work, if they so desire.

References:

[1] Chollet, F. "OpenAI o3 Breakthrough High Score on ARC-AGI-Pub." ARC Prize, 2024.

[2] Akyürek, Ekin, et al. "The surprising effectiveness of test-time training for few-shot learning." Forty-second International Conference on Machine Learning, 2024.

[3] Chollet, F., Knoop, M., Kamradt, G., & Landers, B. (2024). Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604.

[4] Wang, G., Li, J., Sun, Y., Chen, X., Liu, C., Wu, Y., ... & Yadkori, Y. A. (2025). Hierarchical Reasoning Model. arXiv preprint arXiv:2506.21734.

[5] Macfarlane, Matthew V., and Clément Bonnet. "Searching latent program spaces." arXiv preprint arXiv:2411.08706 (2024).

We thank the Reviewer for constructive criticism and thoughtful questions that help us improve the clarity of our paper.

Multiple reviewers had concerns about clarifying where the training objective comes from, what is learned, and what happens to the seeds. As such, we decided that if our paper is accepted, our camera ready version will include three Algorithm boxes in Section 3 to better explain our method:

Algorithm 1: Short program that outputs puzzle P with any solution S. Red text is to be substituted in with hard-coded values produced via Algorithm 2.
---
Set seed_z = <seed_z>;                                                      Hardcoded info about z
Set theta = <theta>;                                                        Hardcoded weights
Set seed_error = <seed_error>;                                              Hardcoded prediction error info
z <- sample(N(0,1), seed_z);                                                Generate inputs z
(puzzle_logits, solution_logits) <- <equivariant_neural_net>(z, theta);     Forward pass
P <- sample(Categorical(puzzle_logits), seed_error);                        Generate puzzle
S <- sample(Categorical(solution_logits), seed_error);                      Generate solution
Return P, S

Algorithm 2: Minimize Description Length.
---
Input: Puzzle P;
Pick equivariant_neural_net architecture dimensions that match up with the data shape of P;
Initialize mu, diagonal Sigma, and weights theta;
foreach step do
    z <- sample(N(mu, Sigma));
    (puzzle_logits, solution_logits) <- equivariant_neural_net(z, theta);
    L <- KL(N(mu, Sigma)||N(0,1)) + crossentropy(puzzle_logits, P);            approximately equal to len(Algo 1) + C
    mu, Sigma, theta <- Adam(grad_mu L, grad_Sigma L, grad_theta L);
end foreach
seed_z <- REC(N(mu, Sigma), N(0,1));                                           Store z with KL(N(mu, Sigma)||N(0,1)) bits
z <- sample(N(0,1), seed_z);
(puzzle_logits, _) <- equivariant_neural_net(z, theta);
seed_error <- REC(Categorical(puzzle_logits), delta_P);                        Store puzzle with crossentropy bits
description <- code for Algorithm 1, with values of seed_z, theta, seed_error,
               and the function equivariant_neural_net substituted in;         Write down Algo 1
Return description

Note that the KL term in the loss measures the length of line 1 in Algorithm 1, and the reconstruction term measures the length of line 3.

Algorithm 3: CompressARC. Same as Algorithm 2, but truncated to only return solution_logits.
---
Input: Puzzle P;
Pick equivariant_neural_net architecture dimensions that match up with the data shape of P;
Initialize mu, diagonal Sigma, and weights theta;
foreach step do
    z <- sample(N(mu, Sigma));
    (puzzle_logits, solution_logits) <- equivariant_neural_net(z, theta);
    L <- KL(N(mu, Sigma)||N(0,1)) + crossentropy(puzzle_logits, P);            approximately equal to len(Algo 1) + C
    mu, Sigma, theta <- Adam(grad_mu L, grad_Sigma L, grad_theta L);
end foreach
Return solution_logits

We will move some of Section 2 to the Appendix to create the space for this.

Clarifying the parts of the method mentioned in weaknesses:

Algorithm 1 above acts as our small description of the puzzle and Algorithm 2 is our proposed way of generating the hardcoded values for Algorithm 1. Algorithm 2 minimizes the description length, and it creates the seed_z and seed_error to hard-code into Algorithm 1. Algorithm 2 also generates puzzle solutions in the middle during it's run as a side effect. So to solve ARC-AGI puzzles, we only need to run it up to that point, which is how we get Algorithm 3, the one we actually run, which learns a mean and covariance together alongside the weights. Algorithm 3 skips the steps of Algorithm 2 that compute seeds, since this is unnecessary to obtain the puzzle solution. This is why seed_error never appears in the implementation.

On the question asked:

In Figure 1, the second-from-the-left boxes represent the input to the equivariant neural network; they are learned tensors just like the weights theta, rather than layers. Even though they get fed as input into the layers, they are also learned parameters alongside the parameters of the layers themselves. The leftmost boxes represent a standard normal distribution. They show that the inputs contribute a loss as we learn them, and that loss is a KL divergence to the standard normal distribution.

On the limitation:

This is 15 iterations per hour if training for all 400 puzzles at once. This is slow because we have been solving the puzzles one at a time, and the GPU is very under-utilized. Due to timing constraints of the rebuttal period, we have so far been unable to produce a working parallelized version to gauge how much faster we might be able to go.

We thank the Reviewer once again for the discussion, and we hope we have addressed their questions satisfactorily. We invite the Reviewer to discuss further on any aspect of our work, if they so desire.

We thank the Reviewer for their thoughful evaluation of our work.

We completely agree with all the strengths and weaknesses listed.

Discussion on weakness #1:

Indeed, our system includes many components that target ARC-AGI specifically, and we are wary that parts of our contribution may not extend outside of this domain. So, we tried our best to focus on the more general, less ARC-AGI specific contributions, by frontloading these main contributions into Sections 1 and 3, and pushing the less transferrable contributions back towards the Appendix. For example, our conversion of MDL into a training algorithm is not specific to ARC-AGI since it can be applied to any kind of data infilling problem. We would appreciate any advice on how to better demarcate the parts of our work that are less targeted at ARC-AGI, that readers can take away and apply to other problems.

A note about weakness #2:

The restriction that our model does not carry information forward between puzzles can conversely be seen as a challenge to tests our system's generalization ability, amplifying the significance of this aspect. Having this inter-puzzle sharing restriction can be seen as either positive or negative depending on how you want to look at it.

Answering Questions:

1. Yes, you are understanding it correctly, the neural network (specifically storing the weights) is highly load bearing, and the differentiable loss that we use is an underestimate. That said, if we presume the weights take up a large constant load (e.g., by storing the weights fully as float32 values), then including/excluding the load of the neural network weights makes no difference to the final inference procedure. This is because the constant added load goes away when differentiating the loss for the training. For MDL purposes, it is not ideal to assume we store the full weights as float32, and if we could efficiently compress them instead as a modification of our current system, this would drastically improve our results. In this direction, there is a possibility to store them in the same format as how we are compressing the neural network inputs, but we expected such a strategy to cause difficulties with posterior collapse in the training dynamics, and did not try this.

2. This work is mostly a proof of concept for the MDL approach towards intelligence. Most of the methodology is not transferrable to other downstream tasks. The main part that is transferrable, is the idea of how to transform the MDL objective into a problem of gradient descent during inference time, as explained in Section 1. This is because the statements of Section 3 still hold when we replace:

   • "solving ARC-AGI puzzles" -> "infilling partially observed data"
   • "incomplete puzzle" -> "observed data"
   • "answers" -> "unobserved data"

   Looking at our method as a data infilling methodology via the above replacements may help other researchers make similar derivations of further methods for other domains like supervised learning and generative modeling.

3. We understand that other studies conclude that model differentiation per puzzle is preferred, and you would like to see any evidence we may have against this to support our ideas for improvement. That said, we do not have early results available for the effects of sharing information between multiple puzzles. In place of this, perhaps we can provide an argument on why we think joint puzzle solving is still a good direction for our work to proceed. Namely, studies which uphold the benefits of model differentiation generally do so from the reference point of having completely shared weights for all puzzles, whereas our system is currently on the other extreme, with completely separate weights for all puzzles. Our best guess is that the best degree of sharing is somewhere in between.

We thank the Reviewer once again for the discussion, and we hope we have addressed their questions satisfactorily. We invite the Reviewer to discuss further on any aspect of our work, if they so desire.