SHARP-Distill: A 68× Faster Recommender System with Hypergraph Neural Networks and Language Models

Essential References Not Discussed

We thank the reviewer for highlighting recent work on LLM-based recommendation and distillation, including DLLM2Rec [1], SAID [2], and POD [3]. These are valuable contributions to this fast-moving area, and we have now incorporated a discussion of them into our Related Work section.

While SHARP-Distill employs a pretrained textual encoder, we clarify that our method is not an LLM-based recommender in the conventional sense. In contrast to DLLM2Rec, SAID, and POD—which rely on full-scale LLMs, prompt tuning, or autoregressive decoding—our framework uses a lightweight review encoder (DeBERTa) on short user-item reviews (~10–40 tokens), avoiding the latency and cost of LLM inference.

To ensure a fair comparison, we evaluated SHARP-Distill against SAID and POD using their official code and settings. DLLM2Rec was excluded only due to lack of code at submission time.

Table 8: Comparison with LLM-based Recommenders (Top-K Metrics and Inference Time)

SHARP-Distill achieves comparable or better accuracy than LLM-based baselines while offering 10–40× lower inference time, thanks to its compact GCN-based student and contrastive fusion strategy. Unlike POD, which requires real-time LLM queries, or SAID, which retains large students, our model delivers LLM-level quality with sub-10ms latency, making it ideal for scalable deployment in time-sensitive environments. We appreciate the reviewer’s suggestion and believe this further clarifies SHARP-Distill’s positioning as an efficient, LLM-alternative framework for real-world recommendation.

Other Strengths and Weaknesses

Thank you for the detailed critique. Due to space limits, we provide a full point-by-point response—including novelty clarifications, differences from [4–6], and positional alignment motivation—at the following link:

🔗 Full Response – Other Strengths & Weaknesses

Theoretical claims have been added to strengthen our framework. Please refer to the following link for details:

🔗 Theoretical Extensions and Supporting Lemmas

We thank the reviewer for the thoughtful feedback and are grateful for the recognition of our work’s strengths, including inference speedup, empirical rigor, and the novel integration of hypergraphs with language models.

Weakness 1) Differentiating RecSys-specific distillation from general graph learning:

(Relation To Broader Scientific Literature)

We appreciate this observation and have clarified the RecSys-specific contributions of our distillation strategy in Section~3.2 of the revised version. In particular:

• User-Item Bipartite Structure Exploitation: Unlike general GNN distillation methods that operate on homogeneous graphs, SHARP-Distill explicitly models the bipartite structure of user-item interactions using dual incidence matrices $\mathcal{H}_U$ and $\mathcal{H}_I$ (Equations 2--3). This formulation captures asymmetric collaborative patterns unique to recommender systems.

• Hypergraph Modeling of High-Order Interactions: Our hypergraph-based formulation extends beyond pairwise user-item links to encode group-level behavioral patterns (e.g., co-engagement with semantically related item clusters), which are crucial for capturing latent preference signals in sparse recommendation data.

• Preference-Aware Contrastive Transfer: As detailed in Equations 17--18, our contrastive objective is designed to align user preference rankings rather than only topological proximity, which is essential for recommendation tasks.

• Multi-Modal Distillation with Reviews: SHARP-Distill uniquely integrates semantic information from textual reviews alongside interaction structures. This dual-source distillation enables robust generalization to cold-start scenarios—an issue more prominent in RecSys than in general graph tasks.

Weakness 2) Comparison with Prior Distillation Methods on Public Benchmarks:

(Essential References Not Discussed)

We have implemented and evaluated SHARP-Distill alongside recent RecSys-specific distillation approaches: UnKD [Chen et al., WSDM 2023] and Graph-less Collaborative Filtering [Zhang et al., NeurIPS 2023]. Both authors released their code, enabling reproducibility on shared datasets.

As shown in Table 2, SHARP-Distill outperforms both in terms of accuracy (P@10 / R@10 / N@10) across Amazon Cellphones, Beauty, and Sports. Moreover, our approach significantly reduces inference time, with speedups of over 2× on average. Due to limited rebuttal-phase time, we could not include large-scale datasets (Amazon CDs, Yelp) for these two models. However, we plan to include these full comparisons in the final version.

Table7: Comparison on Amazon Cellphones, Beauty, and Sports. Metrics: P@10 / R@10 / N@10 (%). Inference time in milliseconds (ms).

While UnKD effectively mitigates popularity bias, it relies on MF/LightGCN backbones and does not incorporate high-order hypergraph structures or textual modalities. SHARP-Distill transfers richer preference-aware signals using contrastive learning from both HGNN structure and DeBERTa-based textual features.

Graph-less avoids structural modeling altogether, trading off explainability and some precision. In contrast, SHARP-Distill uses a lightweight CompactGCN student that distills both structural and semantic knowledge while remaining highly efficient at inference.

Weakness 3) Training efficiency discussion:

Table 6: Teacher Model Training Time (in hours)

These measurements include both HGNN and DeBERTa training and reflect the cost of a one-time offline pretraining phase. For context, we've included baseline LightGCN training times, showing that while our teacher requires approximately 2x longer to train, this cost is amortized through significant inference speedups. Theoretical claims have been added to strengthen our framework.

🔗 Theoretical Extensions and Supporting Lemmas

Claims And Evidence:

We appreciate the reviewer’s thoughtful critique regarding the fairness of our inference-time comparison and broader deployment considerations.

First, we clarify that our comparison is primarily with other distillation-based models (e.g., KRD, LightHGNN), making the results in Table 1 a fair and meaningful evaluation. Regarding teacher model costs, we report the full training time (4.2 hours on a single A100 GPU) in Table 1 (as also noted in Reviewer scTP’s comments). This one-time overhead is amortized over deployment and does not impact end-user latency.

Our inference speed claims (68× faster than HGNN, 40× faster than LightGCN) focus specifically on deployment-time performance, which is critical for real-time recommendation use cases. Moreover, our modular design supports efficient knowledge updates (e.g., retraining only HGNN or DeBERTa if needed), reducing long-term maintenance burdens.

Regarding the concern that KD benefits may diminish over longer training, we conducted extended training experiments (3× standard epochs). We observed stable accuracy across all models, with our relative improvements holding within ±2% variance—indicating that the performance gains are not just due to warm-start effects but reflect genuine enhancements.

We have incorporated these extended training results and deployment discussions into the revised paper. We believe this addresses the reviewer’s concerns while reinforcing our core claim: SHARP-Distill achieves substantial practical gains for real-world recommendation systems.

Questions Q1)

We appreciate the reviewer's insightful question. To assess potential conflicts between semantic and structural features, we conducted an ablation study on the Yelp dataset, evaluating the impact of removing each core component from SHARP-Distill. Specifically, we examined four variants:

• SHARP-Distill: Full model with HGNN-based structure, DeBERTa-based semantics, and contrastive learning
• w/o DeBERTa: Structure only (removes semantic encoder)
• w/o HGNN: Semantics only (removes structural encoder)
• w/o Contrastive Loss: Structure + semantics without alignment

Table 3: Ablation Study on Yelp (P@10 / R@10 / N@10, %)

To quantify the contribution of each component, we calculated the relative performance drop in P@10 with respect to the full SHARP-Distill model:

Δ Performance Compared to SHARP-Distill (P@10):

These results demonstrate the complementary nature of semantic and structural features, and the crucial role of contrastive learning in harmonizing them. Removing DeBERTa causes an 18.8% performance drop, indicating that semantic features are highly informative. Removing HGNN leads to a 24.5% drop, showing the importance of structural signals. Most notably, removing contrastive loss results in a 29.4% performance drop, the largest among all variants. This highlights that simply combining structural and semantic encoders is insufficient—without proper alignment, modality interference reduces effectiveness.

Our contrastive objective (Eqs. 6–7, 17–19) explicitly resolves potential conflicts by aligning heterogeneous modalities into a shared latent space. The observed performance degradation without this alignment empirically validates our hypothesis: contrastive learning not only unifies the two modalities but also unlocks their synergy. These findings are now explicitly stated and discussed in our revised paper.

Questions Q2)

You can find our full response to this comment at the following link:

🔗 Full Response – As discussed in Claims And Evidence section (Reviewer 8pQM)

Theoretical claims have been added to strengthen our framework. Please refer to the following link for details:

🔗 Theoretical Extensions and Supporting Lemmas

We appreciate the reviewer's concerns about preprocessing details and resource analysis. We have added the following information to address these points:

Weaknesses1 and 2. Teacher Model Training Time and Complexity:

As shown in Table 1, our teacher model requires only 4.2 hours to train on a single NVIDIA A100 GPU for the largest dataset (Amazon CDs with 1.2M+ reviews). This one-time offline pretraining cost is modest compared to the significant inference speed benefits gained.

The memory complexity of our HGNN-based teacher model follows the standard pattern of hypergraph neural networks, as discussed in [1]. In a traditional GNN with $L$ layers, $R$ neighbors per node, $d$-dimensional embeddings, and an activation function $f$, the memory requirement typically grows exponentially as $\mathcal{O}(R^L \times d)$. For our specific implementation with the Amazon CDs dataset parameters:

• Using L=3 layers: Memory consumption ≈ 5.67 MB
• Using L=4 layers: Memory consumption ≈ 1,180.16 MB
• Using L=5 layers: Memory consumption ≈ 245,736.24 MB

Our teacher model uses a 3-layer configuration to balance expressiveness and computational efficiency, keeping memory requirements manageable while capturing the necessary high-order interactions.

Preprocessing Details:

Our preprocessing pipeline includes:

• Text normalization (lowercase, punctuation removal, stopword filtering)
• Review text tokenization using DeBERTa's tokenizer with a maximum sequence length of 128
• Construction of user-item interaction hypergraphs where each hyperedge connects a user with all items they've reviewed
• Multi-aspect rating normalization to a [0,1] scale

The preprocessing time is negligible compared to model training time (<10 minutes for the largest dataset) and is a one-time overhead.

Student Model Efficiency:

The distilled student model (CompactGCN) dramatically reduces both the memory footprint and computational complexity:

• Memory requirement: O(R × d) where R is the neighbors per node in the graph
• Practical memory usage: ~0.0268 MB (208×128+128 units)
• Training time: Only 1.2 hours for Amazon CDs (3.5× faster than teacher model)
• Inference time: 9.77ms (68× faster than teacher model, 40× faster than LightGCN)

---

Weakness 3. Multi-component design may complicate implementation:

We appreciate this crucial concern about implementation complexity. SHARP-Distill is intentionally designed with modularity and deployment simplicity as core principles:

A. Clean separation between training and inference: Only the lightweight student model (CompactGCN) is deployed at inference time, eliminating any runtime dependency on complex teacher models.

B. Modular component architecture: Each component—HGNN, DeBERTa, and CompactGCN—connects through clearly defined embedding interfaces, allowing independent optimization.

C. Simplified Implementation Workflow
Our implementation follows a streamlined three-stage process:

1. Teacher Training: We independently train the HGNN model and the pretrained DeBERTa model, which serve as the teacher components.
2. Knowledge Embedding Generation: We extract knowledge embeddings from both trained teacher models.
3. Knowledge Distillation: These embeddings are distilled into the CompactGCN student model using our unified loss function.

[1] Zhang, S., Liu, Y., Sun, Y., & Shah, N. (2022). Graph-less neural networks: Teaching old MLPs new tricks via distillation. ICLR 2022.

Weakness 4. Lacks formal proofs or detailed justification:

You can find our full response to this comment at the following link:

🔗 Full Response – Lacks formal proofs or detailed justification (Reviewer scTP)

Theoretical claims have been added to strengthen our framework. Please refer to the following link for details:

🔗 Theoretical Extensions and Supporting Lemmas