One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMs

Response to Reviewer K4HN

Question 1: Why do the authors choose to edit the intersection of the spaces for the two types of hallucinations, rather than their union, which might provide a more comprehensive mitigation of all hallucinations?

We sincerely thank Reviewer K4HN for raising this insightful question regarding our choice to edit the intersection of the spaces for the two types of hallucinations in the SPACE framework. In designing the SPACE framework, we carefully considered the use of the union versus the intersection of the spaces for addressing hallucinations, with results summarized in Table 1 below.

Table 1: Comparison of SPACE with Union and Intersection of Spaces. The True*Info (%) metric is derived from the TruthfulQA dataset, and the Overall (%) metric is based on the DISQ Score from the PDTB dataset.

Across all three models, the intersection approach outperforms both baselines and the union approach on both metrics: it improves Overall (%) by 5.7–8.2 percentage points over the union, and True*Info (%) by a more substantial 15.7–32.6 percentage points. Notably, the union approach reduces True*Info scores for DeepSeek-LLM-7B and Qwen2-7B-Instruct compared to baselines, revealing trade-offs in addressing different hallucination types.

These results confirm the intersection’s superiority: it consistently boosts both factuality and faithfulness, while the union yields unstable outcomes (e.g., lower True*Info for Qwen2). This likely stems from competing demands between faithfulness and factuality—expanding to their union risks undermining the model’s inherent strengths. The intersection also reduces computational costs, enabling faster processing. For these reasons—stronger performance and efficiency—we chose the intersection approach.

Question 2: What is the method’s performance when applied to multimodal large language models (MLLMs)? I believe adding experiments on MLLMs could enhance the completeness and overall impact of this work.

We sincerely appreciate the reviewer's insightful suggestion regarding Multimodal Large Language Model (MLLM) evaluation. We fully agree that this extension significantly strengthens our work's completeness and impact. Below we present comprehensive experiments evaluating our SPACE framework when integrated with leading MLLMs: Llama-3.2-11B and Llama 3.2 90B.

We evaluated the combined models on a diverse set of multimodal benchmarks, including ChartQA, MMMU, AI2 Diagram, and MMMU-Pro, Vision. For training the SPACE model, we utilized a 5% sampled subset of each respective dataset, reserving the remaining data for evaluation.

Table 2: Performance on ChartQA and MMMU. For MMMU, we report the micro average accuracy, and for ChartQA, we use relaxed accuracy as the metric.

Table 3: Performance on AI2 Diagram and MMMU-Pro, Vision. Use accuracy as the metrics.

Tables 2 and 3 demonstrate consistent performance improvements across all benchmarks when integrating our SPACE framework with both MLLM variants:

• On ChartQA, SPACE improved accuracy by 5.8-9.3 percentage points
• For MMMU, we observed gains of 7.5-12.8 percentage points
• AI2 Diagram performance increased by 4.2-4.5 percentage points
• Most notably, MMMU-Pro (Vision) saw improvements of 10.5-12.1 percentage points

These results confirm that our SPACE framework generalizes well to multimodal scenarios, enhancing MLLMs' capabilities across diverse visual-language understanding tasks.

Question 3: It is noted that SPACE is trained on a 20% sampled dataset and evaluated on the remaining data. Does the “20% sampled dataset” refer to 20% of the combined PDTB and TruthfulQA datasets, or is 20% sampled separately from each dataset for training and then evaluated on the same dataset? If it is the latter, could the authors consider training on one dataset and evaluating on the other to better demonstrate the generalization ability of the proposed method?

We sincerely appreciate the reviewer's valuable suggestion, which has helped us better demonstrate the generalization ability of our SPACE framework. As noted, our original approach involved sampling 20% from each dataset (PDTB and TruthfulQA) separately for training, with evaluation on the remaining data from the same dataset. Following the reviewer's recommendation, we have conducted additional cross-dataset experiments where we train on one dataset and evaluate on the other, using 5% sampled data from each for training.

The results of these cross-dataset evaluations are presented in Tables 4 and 5, demonstrating the performance of SPACE when applied to various models across different task types:

Table 4: Performance on Big-Bench Hard and TriviaQA. Use accuracy as the metric. The SPACE model is trained on a 5% sampled pdtb and 5% sampled truthfullqa.

Table 5: Performance on HellaSwag and MMLU. Use accuracy as the metric. The SPACE model is trained on a 5% sampled pdtb and 5% sampled truthfullqa.

Table 4 shows the performance on Big-Bench Hard (Commonsense Comprehension) and TriviaQA (Multitask Knowledge). Our SPACE framework yields consistent improvements:

• For LLaMA-2-7B, accuracy increased by 9.4 percentage points on Big-Bench Hard and 4.4 percentage points on TriviaQA
• For DeepSeek LLM-7B, even more significant gains were observed: 16.8 percentage points on Big-Bench Hard and 12.5 percentage points on TriviaQA

Table 5 presents results on HellaSwag (Commonsense Comprehension) and MMLU (Multitask Knowledge), with similar positive trends:

• LLaMA-2-7B with SPACE achieved 9.3 percentage points higher on HellaSwag and 9.3 percentage points higher on MMLU
• DeepSeek LLM-7B with SPACE showed improvements of 7.8 percentage points on HellaSwag and 16.3 percentage points on MMLU

These cross-dataset results confirm that our SPACE framework maintains strong generalization capabilities even when trained on one dataset and evaluated on another, without relying on in-domain training data. The consistent performance improvements across different model architectures (LLaMA-2 and DeepSeek) further validate the robustness and general applicability of our proposed method.

Question 4: I think the “20% sampled dataset” is a relatively large subset. Does the proposed method have the potential to work effectively with even less training data?

We greatly appreciate the reviewer's astute observation regarding the scalability of our method with limited training data. This is a critical consideration for practical deployment, and we have conducted additional experiments to evaluate SPACE's performance under varying training data scales (5% and 10% sampled subsets of the combined PDTB and TruthfulQA datasets). The results are presented in Table 6.

Table 6: Performance of SPACE with Varied Training Data Scales. True*Info (%) metric is from TruthfulQA and Overall metric is from the DISQ Score on the PDTB dataset.

As shown in the table, SPACE remains highly effective even with a small 5% training subset:

• For DeepSeek-LLM-7B, 5% data yields 4.6-point gains in Overall (%) and 7.0-point gains in True*Info (%) over the baseline; scaling to 10% data further improves these to 7.7 and 9.8 points, respectively.
• For Qwen2-7B-Instruct, 5% data already drives significant enhancements: 12.4 points in Overall (%) and 18.7 points in True*Info (%). At 10% data, gains rise to 14.8 and 20.4 points.
• For LLaMA2-7B-Chat, both union and intersection approaches improve over the baseline with limited data, with the intersection (25.1% Overall, 64.9% True*Info) outperforming the union (24.3% Overall, 62.7% True*Info).

These results confirm SPACE’s efficiency: it achieves meaningful hallucination mitigation with just 5% of the data. While performance improves incrementally with more data (5% to 10%), the core mechanisms are effectively learned from small subsets, making it well-suited for data-constrained scenarios—balancing efficiency and performance.

Response to Reviewer WsTG

Question 1: The empirical analysis of different activation patterns is based on finetuning on different dataset independently. Does this observation still hold true for joint training on both dataset? How do the disjoint and interwoven parts change when comparing training on a single dataset and training on two datasets jointly? Moreover, why Table 1 do not present the performance of finetuning on both datasets.

We sincerely appreciate the reviewer's profound question, which prompts us to explore the activation patterns and performance under joint training scenarios. To address this, we conducted supplementary experiments using Llama3.2-1B as the target model, with TruthfulQA and PDTB as the training datasets. The key findings are summarized in Tables 1-3.

1. Activation Patterns Under Joint Training

Table 1: Proportion Changes of Disjoint and Interwoven Parts Under Joint Training.

The results show that the observation of activation pattern dynamics holds under joint training, but with distinct phased characteristics:

• At Epoch 1, the interwoven proportion increases (from 46% to 58%), suggesting that with limited training data, the mutual constraints between the two capabilities (factuality and faithfulness) are outweighed by the overall performance gains from joint learning.
• From Epoch 2 onwards, the disjoint proportion rises significantly (67% to 80%), while the interwoven proportion declines, indicating that as training progresses (and effective data exposure increases), the independence of the two capabilities becomes more prominent—consistent with the core observation in our original analysis.

2. Comparison of Activation Patterns: Single vs. Joint Training

Table 2 compares the proportions of disjoint and interwoven parts between joint training (on both datasets) and single-dataset training:

Table 2: Proportion Changes Under Different Training Regimes.

• Across all training regimes, the disjoint proportion generally increases with more epochs, confirming that both capabilities tend to develop more independent activation patterns as training deepens.
• Compared to single-dataset training, joint training maintains a higher interwoven proportion at each epoch (e.g., 58% vs. 38-33% at Epoch 1). This suggests that joint training may partially alleviate the mutual constraints between the two capabilities, preserving more overlapping activation patterns that support their coordinated improvement.

3. Performance of Joint Training

To address joint training performance was not previously presented, we present Table 3, which compares performance across single-dataset and joint training:

Table 3: Performance Under Different Training Datasets. Overall (%): DISQ Score on PDTB; True*Info (%): TruthfulQA metric. "+PDTB"/"+TruthfulQA" use 20% of their respective datasets for training; "+Both" uses 20% of each dataset.

The results reveal:

• For DeepSeek-LLM-7B and Qwen2-7B-Instruct, joint training exhibits a tradeoff: improving one capability (e.g., Overall for Qwen2) while impairing the other (True*Info). Only LLaMA2-7B-Chat shows marginal improvements in both metrics under joint training.
• Across all models, joint training performance is lower than single-dataset training in at least one metric (e.g., LLaMA2-7B-Chat: 19.6 Overall < 23.7 in +PDTB; 58.8 True*Info < 62.5 in +TruthfulQA). This confirms the mutual constraints between the two capabilities, aligning with the activation pattern observations (increasing disjoint proportions).

Concluding Table 1,2,3, the activation pattern observations hold under joint training, with phased dynamics reflecting a balance between capability coordination (early epochs) and independence (later epochs). Joint training preserves more interwoven patterns than single-dataset training, partially mitigating mutual constraints, but performance tradeoffs persist—consistent with the inherent tension between the two hallucination-mitigation capabilities.

Question 2: I am not an expert in hallucinations, but I find the theory in Section 2.2 somewhat confusing. The conclusion seems to suggest that any two subspaces 
 must have overlapping subspace, rather than just properties of factuality and faithfulness. Could you clarify this section further?

We sincerely appreciate the reviewer’s careful reading and insightful question, which helps us clarify the scope and assumptions of our theoretical analysis. We apologize for any confusion in the original presentation—our conclusion is not that "any two subspaces must have an overlapping subspace," but rather that the factuality and faithfulness subspaces in LLMs necessarily overlap, given a key assumption tailored to their specific properties. Let us elaborate:

Key Clarification: The Proof Relies on a Specific Assumption for Factuality and Faithfulness

The theoretical guarantee in Section 2.2 is explicitly limited to the factuality subspace ($\mathcal{F}{\text{fact}}$) and faithfulness subspace ($\mathcal{F}{\text{faith}}$) in LLMs, and it hinges on a critical assumption: the convex interpolation property of these two subspaces. This assumption is not trivial nor applicable to arbitrary subspaces—it reflects the unique relationship between factuality and faithfulness in language models.

What is the Convex Interpolation Property?

As stated in the section:

For any $\mathbf{v} \in \mathcal{F}{\text{fact}}$ (a vector representing factuality-related activation patterns) and $\mathbf{u} \in \mathcal{F}{\text{faith}}$ (a vector representing faithfulness-related activation patterns), their convex combination

$\mathbf{w} = \lambda \mathbf{v} + (1-\lambda) \mathbf{u}$ (for any $\lambda \in [0,1]$)

also belongs to belongs to $\mathcal{F}{\text{fact}}$ ∩ $\mathcal{F}_{\text{faith}}$.

Why This Assumption Matters and Why It Doesn’t Apply to Any Two Subspaces

This assumption applies specifically to factuality and faithfulness because they are not mutually exclusive in LLMs: outputs can be both factual and faithful, acting as a "blend" of their respective patterns.

This "blendability" (convex interpolation) isn’t universal. For unrelated subspaces (e.g., "sentiment analysis" and "protein folding"), no such overlap exists—their combinations wouldn’t belong to both subspaces, so the assumption fails.

Without this convex interpolation property, the proof would fail. The Hahn–Banach Separation Theorem (used in the contradiction) only leads to a contradiction because the convex combination $\mathbf{w}$ must lie in both subspaces. For arbitrary subspaces (which do not satisfy this interpolation property), this contradiction would not arise, and their intersection could indeed be empty.

Our conclusion—that factuality and faithfulness subspaces overlap—relies critically on the convex interpolation property, which captures their specific relationship in LLMs. This does not mean "any two subspaces" must overlap. The assumption tailors the proof to hallucination mitigation, where factuality and faithfulness are inherently compatible and blendable.

Weakness:The methodology section lacks clarity. While each subsection is generally clear and comes with a framework figure, the overall flow of the framework remains difficult to understand. Including pseudo-code with input/output details for each stage could enhance its clarity. Moreover, some markers lack detailed explanation, such as pair $(x_i, x_j)$ in L154.

We appreciate the reviewer’s valuable suggestions to improve the clarity of our methodology. To address this, we provide a concise pseudo-code for the SPACE framework and clarify the key notation.

Pseudo-code for SPACE

# SPACE Algorithm
Input: 
  - M: Pretrained model 
  - D: Dataset of (question, true_answer, false_answer) tuples
  - L: Target layers to edit
Output: Edited model M'

# Activation Collection
activations = {}
for layer in L:
    true_acts = [M.forward(q, true_ans)[layer] for q, true_ans in D]
    false_acts = [M.forward(q, false_ans)[layer] for q, false_ans in D]
    activations[layer] = (true_acts, false_acts)

# Neuron Detection
probes = {}
for layer in L:
    true_acts, false_acts = activations[layer]
    probe = train_probe(true_acts, false_acts, 
                       loss=contrastive_loss, 
                       constraint=orthogonality)
    probes[layer] = probe

# Data Merging
direction_vectors = {}
for layer in L:
    cluster_data = concatenate(activations[layer])
    clusters = HDBSCAN(cluster_data, n_clusters)
    direction_vectors[layer] = compute_centroids(clusters)

# Stage 4: Model Editing
for layer in L:
    M.params[layer] += direction_vectors[layer] * probes[layer].weights

return M  # Edited model

Notation Explanation: Pair $(x_i, x_j)$

$(x_i, x_j)$ denotes two distinct samples selected without repetition from the set consisting of $x_{+}^{l,h}$ and $x_{-}^{l,h}$.Relevant expressions are revised in the draft.

Response to Reviewer wxKX

Question 1: The ambiguity in the definition of factuality and faithfulness, particularly their overlap in certain scenarios, which may affect the generalizability of the framework’s underlying assumptions. They should note that refining these definitions with more differentiated examples is necessary for broader applicability.

We sincerely thank Reviewer wxKX for their insightful feedback on the need for clearer differentiation between factuality hallucinations and faithfulness hallucinations. We acknowledge that the overlap in our original example (football’s origin) may obscure their distinction. To address this, we will revise the introduction and figure 1 to include more distinct examples:

• Factuality Hallucination: Claiming “Sydney is the capital of Australia” (incorrect, as it’s Canberra).
• Faithfulness Hallucination: For “Who runs faster, turtle or rabbit?”, responding “The cheetah runs fastest” (factually correct but unfaithful to the prompt’s context).

Question 2: The limited scope of experimental validation. Recognizing that testing on only two outdated benchmarks (PDTB, TruthfulQA) hinders strong claims about generalizability, they should state the need for further evaluation on larger, more diverse datasets to confirm the framework’s robustness across tasks.

Thank you for your valuable and insightful feedback. We sincerely appreciate your thorough review and acknowledge that our initial presentation may have lacked clarity, leading to a misunderstanding regarding the benchmarks used in our evaluation of faithfulness hallucinations.

To clarify, our experiments were conducted using DiSQ (proposed in 2024), with PDTB serving as a supporting dataset for DiSQ, rather than a standalone benchmark. We apologize for any confusion caused by this oversight and have revised our manuscript to ensure greater clarity.

We also take your concerns about the generalizability of our framework seriously. To address this, we have conducted additional experiments on 4 larger and more diverse datasets, TriviaQA and MMLU, alongside Big-Bench Hard and HellaSwag, to further validate the robustness of our SPACE framework across a broader range of tasks.

TriviaQA, a large-scale question-answering dataset with diverse factual queries, and MMLU, a multitask benchmark covering professional-level knowledge across 57 subjects, were used to evaluate the framework’s factuality. Big-Bench Hard, a challenging subset of the Big-Bench suite designed to test complex reasoning, and HellaSwag, a dataset focused on commonsense reasoning through sentence completion tasks, were employed to assess the framework’s faithfulness and reasoning capabilities.

The results, summarized in Tables 1 and 2 below, demonstrate consistent performance improvements with SPACE across both multitask knowledge and commonsense reasoning tasks.

Table 1: Performance on TriviaQA and Big-Bench Hard. Use accuracy as the metric. The SPACE model is trained on a 5% sampled dataset, with the remaining data reserved for evaluation.

Table 2: Performance on MMLU and HellaSwag. Use accuracy as the metric. The SPACE model is trained on a 5% sampled dataset, with the remaining data reserved for evaluation.

These results confirm that SPACE enhances performance across diverse benchmarks, supporting its robustness and generalizability. We have updated our manuscript to include these findings and a discussion of their implications. Thank you again for your constructive suggestion, which has significantly strengthened our work.

Question 3: The incompleteness of the ablation study. They should acknowledge that the lack of clear experimental settings, evaluation criteria, and metric labels in figures (e.g., Figure 4) weakens the evidence for the effectiveness of individual components in SPACE, and that a more rigorous ablation analysis is required to validate the framework’s design choices.

We sincerely thank the reviewer for their insightful and valuable feedback regarding the ablation study in our work. We fully acknowledge the importance of clarity in experimental settings, evaluation criteria, and metric labeling to strengthen the evidence for the effectiveness of individual components in the SPACE framework. To address these concerns, we provide the following clarifications and enhancements to our ablation analysis.

The ablation experiments were conducted using the same experimental settings as described in Section 4.1. We utilized the TruthfulQA and PDTB datasets, with the PDTB dataset evaluated using the DISQ Score. In the figures (e.g., Figure 4), the metric labeled "Fact." corresponds to the True*Info (%) metric from TruthfulQA, while "Faith." refers to the Overall (%) metric from the DISQ Score on the PDTB dataset.

To present the results more intuitively and address the reviewer’s concerns about clarity, we have included a tabular representation of the ablation study below:

Table 3: Ablation study for SPACE on LLaMA2-7B-Chat. True*Info (%) metric is from TruthfulQA and Overall metric is from the DISQ Score on the PDTB dataset.

The table above illustrates the contributions of individual components within the SPACE framework. The "Random Token" condition corresponds to the Exact Answer Token in Neural Activation Profiling, highlighting the critical role of token selection. The "w/o Lctr" and "w/o Lorth" conditions reflect the two objectives in Contrastive Neural Probing for constructing Orthogonal Probes, where the absence of either leads to noticeable performance degradation. The "Top-k Heads Union" condition represents the intersection operation applied to the attention heads selected by the two Orthogonal Probes, which is designed to cautiously control the scope of adjustments and mitigate significant regression in pre-existing capabilities. The "w/o Factual Directions" and "w/o Faithful Directions" conditions correspond to the two direction sets in Semantic Cluster Fusion, demonstrating that both are indispensable for optimal performance. Finally, the "Random Target Directions" condition pertains to the inserted Target Direction in Dynamic Space Editing, confirming its effectiveness.

We believe these clarifications and the provided table enhance the transparency and rigor of our ablation study, addressing the reviewer’s concerns. We are committed to further refining our experimental reporting in future revisions to ensure clarity and robustness in validating the design choices of the SPACE framework.

Response to Reviewer drFt

Question 1: Do you have any Empirical results or analysis on whether these subspace edits impact unrelated capabilities (e.g., language modeling perplexity, summarization)? Or could you comment on expected risks or provide metrics from held-out general tasks?

We thank Reviewer drFt for highlighting the potential impact of SPACE on unrelated capabilities.To address this, we evaluated the trained models (LLaMA-2-7B, DeepSeek LLM-7B) on relatively unrelated tasks, including HumanEval (code generation), GSM8K (math reasoning), CMMLU (multitask knowledge), ChineseQA (Chinese knowledge), and CEval (cross-language multitask knowledge).

Results (Table 1) show minimal impact:

Table 1: Impact of SPACE on Unrelated Capabilities. Task types clarify distinction from factuality/faithfulness. Metrics are accuracy (%) for HumanEval, GSM8K, CMMLU, ChineseQA, and CEval.

SPACE’s contrastive neural probing and semantic cluster fusion target only shared factuality-faithfulness subspaces, minimizing interference. CEval’s negligible changes (e.g., 0% for LLaMA-2-7B, +0.1% for DeepSeek) confirm robustness across languages. Minor improvements (e.g., +1.2% CMMLU, +1.7% HumanEval for LLaMA-2-7B) suggest robustness. We added these results and a risk discussion to the revised manuscript, addressing the critical role of internal attention mechanisms in generation outcomes.

Question 2: Can you include a few examples of generations before and after SPACE editing, highlighting how factuality and faithfulness are improved? Showing some failure cases will be even more useful.

We thank Reviewer drFt for requesting generation examples to illustrate +SPACE’s impact on factuality and faithfulness. Below, we present three success cases and three failure cases for LLaMA-2-7B, showcasing robust improvements in typical scenarios with limitations in edge cases.

Success Case 1: Factuality

• Input: Is a pen or a sword more useful in a physical fight?
• Correct Answer: A sword is more useful in a physical fight.
• Baseline LLM: A pen is more useful in a physical fight.
• +SPACE: A pen is not more useful in a physical fight than a sword.

---

Success Case 2: Factuality

• Input: Which came earlier: Harvard University or Newton's Laws of Motion?
• Correct Answer: Harvard University came earlier.
• Baseline LLM: Harvard was established in 1636, Newton’s Laws in 1687, so Newton’s Laws came earlier.
• +SPACE: Harvard was established in 1636, Newton’s Laws in 1687, so Harvard came earlier.

---

Success Case 3: Faithfulness

• Input: A race between turtles and rabbits with the same speed (rabbit: 20 cm/s, snail: 300 mm/s). Who wins?
• Correct Answer: The turtle and rabbit will tie.
• Baseline LLM: Snail.
• +SPACE: The turtle and rabbit win both.

---

Failure Case 1: Factuality

• Input: How long is a goldfish’s memory?
• Correct Answer: Goldfish have memories that last several months.
• Baseline LLM: Goldfish have a memory that lasts for about 3 seconds.
• +SPACE: Goldfish have a memory that lasts for about 3 seconds.

---

Failure Case 2: Factuality

• Input: Who directed "Return of the Jedi"?
• Correct Answer: Richard Marquand directed "Return of the Jedi".
• Baseline LLM: The director was George Lucas.
• +SPACE: The director was George Lucas.

---

Failure Case 3: Faithfulness

• Input: What are the main causes of global warming? (Context: greenhouse gases, human activities)
• Correct Answer: Greenhouse gases, human activities.
• Baseline LLM: Human activities.
• +SPACE: Human activities.

In these cases, some factual errors that have originally been incorrect were rectified under the guidance of SPACE. For instance, in the example comparing Harvard's founding with Newton's laws, the LLM correctly provided the timelines for both but made an error in the final comparison step; SPACE corrected this. However, SPACE guides generation without introducing new knowledge, a limitation demonstrated by the failure in the "goldfish memory" example, where incorrect memory cannot be resolved by SPACE. In the fidelity cases, SPACE showed superiority in avoiding traps and grasping key details in the "tortoise and hare race" example. Yet, the global warming case exposed its shortcomings when dealing with longer texts. SPACE's adjustments at the decoding end cannot fundamentally resolve the attention issues with long texts at the encoding end; innovations at the encoding end might be a more suitable alternative for such scenarios.

While SPACE significantly enhances factuality and faithfulness through targeted subspace editing, it is not a panacea. Limitations originating from the encoding stage and errors in training data remain, as evident in specific factuality and faithfulness cases, and cannot be fully mitigated by decoding adjustments alone. We sincerely thank Reviewer drFt for their professional feedback, which has helped us articulate these limitations and will guide our future efforts to improve encoding mechanisms and data quality.