Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains

Dear Reviewer #MsBS,

Thank you for acknowledging CoLaR’s novelty and empirical breadth, and for enumerating the remaining gaps in domain coverage, scale, dataset, comparative baselines, algorithmic justification, and explainability. We address each concern below.

W1.1: Clarification of GSM8k. Thank you for pointing this out. There are three versions of GSM8k: 1) GSM8k, the original one, where there are 8k samples with natural language reasoning chains, and 2) GSM8k-Aug, the one we used for training, which has extended training samples (~385k) and condensed reasoning chains, and 3) GSM8k-Hard, the one we used for OOD testing, where the numbers are in larger range and harder to compute. The test set of GSM8k and GSM8k-Aug are the same. We will clarify this in our final revision.

W1.2 & W2 & W3: Scaling-up, new baseline, and non-math reasoning tasks. Yes, CoLaR is capable to scale up to large foundation models, generalize to non-mathematical reasoning domains, and CoLaR surpasses TokenSkip on MATH dataset with fewer reasoning steps.

Appendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26~40% (1B) to 43~61% (8B) while retaining comparable compression ratios.

To extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on physics, chemistry, and biology:

(The reported results of GSM and MATH are slightly different from Table 1 of our manuscript. This is because we are using a different version datasets, GSM8k instead of GSM8k-Aug, to align with TokenSkip. We are unable to directly compare our CoLaR with LightThinker, which mainly focuses on replacing already-generated cot tokens with dense representations to reduce memory peak.)

Four patterns emerge:

• First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.
• Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.
• Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size.
• Finally, CoLaR-8B surpasses TokenSkip by a large margin on SFT and RL version with fewer reasoning steps on MATH dataset.

W4: Reasons to use GRPO. Thank you for this question. GRPO is selected because the task is a canonical exploration–exploitation dilemma: for each question a combinatoric space of latent reasoning trajectories exists, yet only the subset that yields the correct answer is admissible.

GRPO instantiates this balance explicitly: it first explores by sampling a group of complete latent chains, then exploits by reinforcing those that are both correct and minimal in length.

Moreover, in contrast to PPO, GRPO dispenses with a learned value function and instead relies on the verifiable reward alone, reducing variance and computational overhead while preserving principled policy optimization.

We will add relevant discussion to Section 3.3 in our final revision.

W5.1: Generalization to non-integer compression factors. As acknowledged in the Limitations Section, CoLaR’s reliance on discrete prompting precludes fine-grained, non-integer compression ratios.

We propose to address this limitation via a stochastic training strategy that blurs compression factor boundaries: at each forward pass, we randomly merge either one or two consecutive tokens, thereby inducing an effective compression factor that fluctuates around 1.5.

This strategy approximates continuous control while remaining fully compatible with the model’s discrete tokenization, and we will empirically validate its efficacy in our future work.

W5.2: Explainability. We thank the reviewer for this concern. Appendix B, Figure 5 demonstrates that CoLaR’s latent variables—compressed token embeddings—can be faithfully decoded: a simple cosine-similarity search against the token embedding matrix reliably recovers human-readable fragments such as “21 / 7 = 3 × 5 = 15.”

More broadly, recent studies [1] reveal that explicit CoT steps can diverge from the model’s actual decision process or harbor silent biases; verbalization is therefore not a guarantee of transparency. Retrieving embeddings from CoLaR offers the same level of interpretability as CoT, while being more concise.

Finally, we argue that human thought itself resides in a latent space. Russian distinguishes голубой (“sky-blue”) from синий (“ocean-blue”)—concepts with no exact English counterpart—yet speakers share the same perceptual embedding in their minds. An old adage captures this: “It can be grasped, yet cannot be spoken.” Echoing Wittgenstein, we believe much of human reasoning unfolds in a non-linguistic, latent form; CoLaR merely gives us direct access to that space and lets us translate it back into words when necessary.

[1] Barez, Fazl, et al. "Chain-of-thought is not explainability." Preprint, alphaXiv (2025): v2.

Your insightful remarks have been invaluable to the refinement of our work; we stand ready to address any additional concerns you may have.

Authors of CoLaR #12942

Dear Reviewer #nEhT,

We sincerely appreciate your recognition on our latent reasoning framework along with RL pipeline, and highlighting the compression robustness, termination clarity, and scalability concerns. We address each point below.

W1: Generalization of the averaging operation. Although the compression module is intentionally lightweight, its robustness is inherited from the backbone LLM.

In CoLaR, every latent embedding is merely a scaled linear combination of in-vocabulary token embeddings. Because the LLM was trained on massive, diverse corpora, the probability mass over tokens is already smooth; extreme outliers that would make the average OOD are therefore rare. Empirically, we observed no stability issues across three OOD datasets (GSM-hard, SVAMP, and MultiArith), indicating that the exact form of the compression is not a brittle design choice.

W2 & Q1: Termination mechanism and overhead. CoLaR decides when to stop via the Language Head, not a fixed length.

At every latent step we feed the final hidden state into the Language Head and sample the next token; if the token is <end-of-thinking>, we switch to answer generation. This is identical to how any autoregressive LLM ends a turn, so the extra compute is a single linear layer (hidden-size × vocab-size) forward pass.

To quantify the cost, we timed 100 latent reasoning steps on GSM at batch-size 32 on one A100 GPU: LLM layers took 10.8 s, while the Language-Head check added only 0.6 s (5% of total). Replacing the full Language Head with a Tiny Linear layer (by extracting the <end-of-thinking> token column of the Language Head) keeps accuracy at 26.7% and cuts the check to 0.3 s—noise compared with the 3–10× latency savings we already obtain via shorter chains.

W3: Scaling-up and domain generalization. Yes, CoLaR is capable to scale up to large foundation models, and generalize to non-mathematical reasoning domains.

Appendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26~40% (1B) to 43~61% (8B) while retaining comparable compression ratios.

To extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on physics, chemistry, and biology: (The reported results of GSM and MATH are different from Table 1 of our manuscript. This is because we are using a different version of datasets and prompts to align with another baseline method)

Three patterns emerge:

• First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.
• Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.
• Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size.

We thank you for your thorough review and look forward to any further questions or suggestions that could strengthen the paper.

Authors of CoLaR #12942

Dear Reviewer #VtEU,

We are grateful for your recognition of our dynamic-compression design, rigorous evaluation, and the documented RL-driven gains. Below we address the four remaining weaknesses that you precisely articulated.

W1: Efficiency vs. accuracy in practice. While CoLaR currently lags CoT on raw accuracy, real-world deployments already trade a few percentage points for 2–10× cheaper inference (think Gemini-Flash vs. Gemini-Pro). CoLaR’s carbon footprint drops proportionally with compression factor, making the trade-off attractive for latency-sensitive or eco-conscious applications.

W2 & Q1-Q2: Scaling to larger models & Non-math datasets. Yes, CoLaR is capable to scale up to large foundation models, and generalize to non-mathematical reasoning domains.

Appendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26~40% (1B) to 43~61% (8B) while retaining comparable compression ratios.

To extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on physics, chemistry, and biology: (The reported results of GSM and MATH are different from Table 1 of our manuscript. This is because we are using a slight different version of datasets to align with another baseline method TokenSkip.)

Three patterns emerge:

• First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.
• Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.
• Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size.

W3: Explainability. We thank the reviewer for this concern. Appendix B, Figure 5 demonstrates that CoLaR’s latent variables—compressed token embeddings—can be faithfully decoded: a simple cosine-similarity search against the token embedding matrix reliably recovers human-readable fragments such as “21 / 7 = 3 × 5 = 15.”

More broadly, recent studies [1] reveal that explicit CoT steps can diverge from the model’s actual decision process or harbor silent biases; verbalization is therefore not a guarantee of transparency. Retrieving embeddings from CoLaR offers the same level of interpretability as CoT, while being more concise.

Finally, we argue that human thought, or part of human thought, resides in a latent space. Russian distinguishes голубой (“sky-blue”) from синий (“ocean-blue”)—concepts with no exact English counterpart—yet speakers share the same perceptual embedding in their minds. An old adage captures this: “It can be grasped, yet cannot be spoken.” Echoing Wittgenstein, we believe much of human reasoning unfolds in a non-linguistic, latent form; CoLaR merely gives us direct access to that space and lets us translate it back into words when necessary.

[1] Barez, Fazl, et al. "Chain-of-thought is not explainability." Preprint, alphaXiv (2025): v2.

W4: Closing the accuracy gap. Thank you for this insightful question. CoLaR’s SFT stage only distills existing CoT paths, so its ceiling is inevitably “CoT-minus-distillation-loss.” Reinforcement learning, however, lets the model discover shorter latent paths that can outperform the teacher: CoLaR-8B-RL already beats CoT-8B on GPQA and narrows the gap on MATH (52.4 vs 54.6). We expect continued RL at larger scales to erase or reverse the deficit, just as R1 surpassed its CoT teacher. We will add more detailed discussion in Section 4.2 of our final revision.

Thank you for your constructive observations; we would be glad to address any additional concerns you may have.

Authors of CoLaR #12942

Dear Reviewer #dukW,

We thank you for highlighting both the merits of our probabilistic-latent design and the concerns regarding baseline fidelity and long-chain evaluation. We address each point below.

Q1: Baseline implementation details. We strictly followed the original papers and release notes of iCoT and Coconut, applying identical base settings (frozen Llama-3.2-1B-Instruct, LoRA rank 128, batch size 256, learning rate 1e-4, max 50 epochs) and dataset splits; their custom hyper-parameters are documented in line 228 and will be expanded in the final revision.

W1: Lower baseline scores on a larger backbone. The drop is expected: LoRA fine-tuning exposes only 13.6 M parameters on Llama-3.2-1B, whereas the GPT-2 baselines in prior work train all 117 M parameters at FP32 precision:

The table shows that when we replicate CoLaR on full-parameter GPT-2 its GSM accuracy rises from 26.8% to 35.2%, confirming that parameter count—not backbone capacity—dominates on these short chains. As anticipated, this gain sacrifices out-of-domain generalization (SVAMP and MultiArith drop sharply), so we kept the LoRA setting for fair comparison.

W2 & Q2: Applicability to long-reasoning models. Thank you for highlighting the gap in long-chain settings. We agree that compressing truly long reasoning traces is where CoLaR can deliver the most value, and we have already taken the first step in that direction: on DeepSeek-R1-Distill-Qwen-1.5B, we trained and evaluated CoLaR on the MATH dataset whose average reasoning chain is ~200 tokens—ten times longer than GSM’s. As shown in the paper (Table 2), CoLaR still achieves an 82.8% reduction in chain length with a 5.36% accuracy gain after RL, demonstrating that the method scales gracefully to longer contexts.

What we have not yet attempted is compressing chains of several thousand tokens, such as those generated by the full DeepSeek-R1 671 B model. The bottleneck is purely computational: training on such long sequences would require either (1) multi-node GPU clusters to fit the enlarged teacher traces, or (2) an additional “latent pre-training” phase where we first distill very long CoT paths into compressed embeddings offline, then fine-tune the latent head. Both are in our immediate roadmap.

We are optimistic about scaling the latent reasoning length: because CoLaR’s next-compressed-embedding objective is structurally identical to next-token prediction—the paradigm that has already scaled to million-token contexts.

We sincerely appreciate your meticulous feedback and remain available for any further clarifications.

Authors of CoLaR #12942

Dear Reviewer #dukW,

Thank you again for your constructive and detailed review. Your comments have been very helpful in identifying points where the paper can be strengthened.

In our final revision, we will:

• elaborate the baseline settings more clearly in Section 4.1;

• add a GPT-2–based model for more direct comparison with prior work;

• extend Section 4.2 with additional datasets and results from 1B–8B models.

Looking ahead, we plan to explore longer latent reasoning chains, including latent pre-training with extended compute resources. Your observations on this point were both insightful and motivating.

We would also like to share: after the discussion period, other reviewers have positively revised their evaluations. We truly value your thoughtful assessment and the role it has played in shaping our final improvements.

Best regards,

Authors of CoLaR #12942