Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning

Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.

$\bullet$ Q1: We have updated Section 3.3 to clarify our structural‑information notation by explicitly defining $\alpha^-$ as the parent of any non‑root tree node $\alpha$.

$\bullet$ Q2: For Line 182, we determine $k$ by sweeping over all plausible values in $[1, |\mathcal{S}|]$, constructing for each state vertex a $k$‑nearest‑neighbor graph (keeping its $k$ most similar neighbors) and computing the resulting one‑dimensional structural entropy. We then select the $k$ that maximizes this entropy. A pseudocode description will be provided in Appendix A of the camera‑ready version after conference acceptance.

As for Line 184, $\mathcal{K}$ denotes the maximum depth of both the coding tree and the diffusion hierarchy. While HCSE optimizes the partition structure under a fixed depth limit, it does not choose that limit itself. Accordingly, we specify $\mathcal{K}$ upfront in our experiments, with detailed justification and a sensitivity analysis presented in Appendix C.1.

$\bullet$ Q3: Indeed, our trajectory segmentation is driven by the automatically discovered state communities rather than fixed time intervals, so the resulting sub‑trajectories naturally vary in length. To train the diffusion model—which requires inputs of uniform length—we pad each shorter subsequence with terminal states until it reaches the prescribed length (see lines 483–485 in Appendix A.6). Furthermore, when $h=1$, each trajectory segment corresponds to an entire sequence of state–action pairs (not just a single pair), as explained in the text.

$\bullet$ Q4: Here’s a toy example of how SIHD hierarchically partitions a trajectory of 1000 timesteps into sub‑trajectories at different levels $h$. In each sub‑trajectory, the final state is treated as that segment’s ``sub‑goal."

Level $h=\mathcal{K}$ (top, coarsest): $[1, 2, \dots, 1000]$.

Level $h=\mathcal{K} - 1$: $[1, 2, \dots, 90], [91, 92, \dots, 200], \dots, [901, 902, \dots, 1000]$.

Level $h=2$ (fine): $[1, 2, \dots, 30], \dots, [78, 79, \dots, 90], \dots, [910, 941, \dots, 950], \dots, [968, 969, \dots, 1000]$.

Level $h=1$ (finest): $[1, 2, \dots, 8], [9, 10, \dots, 15], \dots, [988, 989, \dots,994], [995, 996, \dots, 1000]$.

$\bullet$ Q5: In Figure 3(c), $\mathcal{K}-1$ ndeed represents an arbitrary intermediate layer and can be equivalently replaced with $h$. We have revised Figure 3 accordingly to improve clarity.

Additionally, we have updated the main text prior to line 197 to explicitly introduce the subtrajectory segmentation and subgoal extraction for the second-highest layer, as well as the parameter $\tau_g^{h,i}$, which denotes the subgoal trajectory length. To further address your comment, we now clarify the notation $\tau^\mathcal{K}_g$ and $\tau_g^{h,i}$ in the corresponding context to ensure they are well-defined before use.

We agree that including the trajectory partitioning example you suggested would enhance understanding. We will incorporate it into the appendix in the camera-ready version.

$\bullet$ Q6: During inference with the highest-level diffusion model, we employ classifier-free guidance by setting the reward signal—used as conditioning during training—to null. This allows the model to generate a sequence of subgoals without explicit reward input. For conditioning the next-level diffusion model, we select the next subgoal from this sequence and use it to compute the corresponding conditional signal.

Specifically, we locate the node in the optimal encoding tree—constructed from offline trajectories—that is closest to the subgoal state in Euclidean space. Based on this matched node (i.e., the state community), we compute the conditioning signal according to Equation 12. This process is repeated hierarchically for each layer until the full trajectory is generated.

$\bullet$ Q7: Here, ``shared" refers to the fact that, for a given hierarchy level $h$, we train a single diffusion model whose parameters are shared across all sub‑goal sequences at that level. In other words, each layer $h$ indeed has its own diffusion network, but within that layer every sub‑trajectory’s generation uses the same model weights—i.e., the model is shared across sub‑goals at level $h$, rather than having a distinct network for each individual segment.

$\bullet$ Q8: Thank you for the suggestion. We have added citations to Classifier‑Free Diffusion Guidance [1] and More Control for Free! Image Synthesis with Semantic Diffusion Guidance [2] at line 230.

$\bullet$ Q9: To compute the joint probability $p_{\theta_1}(s_t, s_{t+1})$ between any adjacent state pair, we proceed as follows: we begin by sampling Gaussian noise sequences $\tau_{g, K}^{1,1} \sim \mathcal{N}(0, \mathcal{I})$. These represent initial noisy trajectories. We then apply the reverse process described in Equation 11 to iteratively denoise these trajectories, producing $n$ predicted sequences $\tau_{g, 0}^{1,i}$ with $i=1, \dots, n$. From each denoised sequence, we extract the states at time steps $t$ and $t+1$, forming a sample set $\\{s_t^i, s_{t+1}^i\\}$. Finally, we use a two-dimensional Gaussian kernel density estimator over this sample set to estimate the joint distribution $p_{\theta_1}(s_t, s_{t+1})$.

For clarity and reproducibility, we will include the complete procedure for joint probability estimation as pseudocode in Appendix A, after conference acceptance.

$\bullet$ Q10: As explained previously, constructing the weighted state graph $\mathcal{G}_s^\prime$ requires estimating the joint probability between all adjacent state pairs in the offline trajectories, which is computationally expensive. To mitigate this overhead, we derive in Theorem 4.2 a lower bound of the structural entropy in the form of Shannon entropy. This formulation not only provides theoretical justification for the entropy regularization but also yields an efficient estimation strategy.

Specifically, we leverage the $n$ denoised sequences $\\{\tau_{g, 0}^{1,i}\\}$ obtained from the base diffusion model $\epsilon_{\theta_1}$. The states $s_t^i$ from these sequences are hierarchically clustered according to their Euclidean distances to the layer-wise centroids of the encoding tree $\mathcal{T}_s^*$, yielding community sets $\\{\mathcal{U}_h\\}$ with $h=1,\dots,\mathcal{K}-1$. For each community set $\mathcal{U}_h$, we approximate its Shannon entropy $\mathcal{H}(\mathcal{U}_h)$ using a $k$-nearest neighbors entropy estimator [3], which averages the Euclidean distances between each community and its $k$-th nearest neighbor, as detailed in Equation 15.

Since the entire entropy computation is based solely on the lowest-level diffusion model $\epsilon_{\theta_1}$, the entropy regularization is applied only at $h=1$. This design choice ensures computational efficiency while still enforcing meaningful structure through the regularization.

$\bullet$ Q11: We have added a description of the key hyperparameter settings in Appendix A and will ensure that this information is included in the camera-ready version upon acceptance.

$\bullet$ Q12: All experiments in our work were conducted on five Linux servers, each equipped with an NVIDIA RTX A6000 GPU and an Intel i9-10980XE CPU running at 3.00 GHz. To reduce redundant computation, we performed state community partitioning offline and stored the resulting structures in a dictionary format. This design ensures that the community assignments do not need to be recomputed during training or inference, thereby maintaining the overall computational feasibility of training and deploying the hierarchical diffusion model on the D4RL dataset.

$\bullet$ Q13: Intuitively, increasing the number of layers in a hierarchical diffusion model would introduce additional time overhead. As shown in Tables 4 and 5, SIHD generally requires more training and inference time than the two-layer HD model in most scenarios. However, it is important to note that under a fixed time horizon, adding more hierarchical layers leads to shorter sequence lengths for each individual diffusion model. Furthermore, both training and inference are implemented with parallel computation across layers. As a result, the time overhead introduced by the hierarchical structure is significantly mitigated, and in some cases, SIHD even demonstrates slightly better time efficiency compared to HD.

$\bullet$ Q14: In Figure 2, the term ``online decision-making" refers specifically to the use of the trained hierarchical diffusion model for online inference or trajectory planning, rather than interaction-based online reinforcement learning. To avoid any potential misunderstanding, we have revised Figure 2 accordingly to clarify this intent.

References:

[1] Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.

[2] More Control for Free! Image Synthesis with Semantic Diffusion Guidance. IEEE/CVF 2023.

[3] Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 2003.

input: a weighted, undirected, and complete graph $\mathcal{G}_s$
output: a sparsified graph $\mathcal{G}_s$
for $k = 1$ to $\min {|\mathcal{S}|, 20}$ do
  if the $k$NN graph exists then
      calculate the one-dimensional structural entropy $\mathcal{H}^1(\mathcal{G}_s)$ for $k$ 
  end if
end for
select the optimal value k by minimizing $\mathcal{H}^1(\mathcal{G}_s)$

Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.

$\bullet$ W1: To improve clarity and readability, we have made comprehensive revisions to Section 4 of the paper. Specifically, we added clear explanations for the subscripts and superscripts used in our notation—where superscript $h$ and $i$ indicate the diffusion hierarchy level and temporal ordering within that level, and subscripts $g$ and $sa$ refer to subgoal sequences and state-action subtrajectories, respectively.

In response to your comments on Figures 2 and 3, we redesigned Figure 2 to visually separate the three core modules into distinct subfigures, each illustrating the technical workflow of one stage in a more intuitive manner. Additionally, we expanded Figure 3 to provide a more concrete, visual depiction of the full hierarchical diffusion process—from community partitioning, trajectory segmentation, and subgoal extraction, to conditional signal definition—based on offline trajectories.

$\bullet$ W2: In Equation 8, within the control-as-inference paradigm, the subgoal $g_i^{\mathcal{K} - 1}$ is implicitly defined as the terminal state of the corresponding subtrajectory. Specifically, it is set to the cumulative length of preceding subtrajectories, i.e., $g_i^{\mathcal{K} - 1} = \sum_{j=1}^{i} l_{sa}^{\mathcal{K} - 1,j}$. This dependence is not explicitly shown on the right-hand side of Equation 8 but is encoded through the structure of the generated subtrajectory $\tau_g^{h,i}$.

To generalize this idea, Theorem 4.1 formalizes a unified formulation of hierarchical conditional diffusion, where the conditioning term $y(\tau_g^{h,i})$ represents a flexible signal associated with the subgoal state. In prior approaches, this is often instantiated as the cumulative reward over the subtrajectory. However, such a reward-based signal is often insufficient—especially in long-horizon or sparse-reward settings—due to its limited informativeness and locality.

To address this, in our framework:

At the top layer, where the receptive field covers the full trajectory, we retain cumulative return-based conditioning to reflect global objectives.

At intermediate layers, where the model operates on shorter temporal windows with limited context, we introduce a structure-aware signal based on the expected hitting probability of subgoals via random walks on the offline state graph. This formulation, defined in Equation 12, captures meaningful topological relationships in the latent space and serves as a robust and informative form of guidance for lower-level diffusion.

We believe this hierarchical conditioning strategy is key to the improved performance and flexibility of our model, and we will clarify these connections more explicitly in the final version of the paper.

$\bullet$ W3: We acknowledge that introducing additional hierarchical levels may increase the computational burden during both training and inference. To address this, we included a detailed analysis in the original manuscript (Appendix C.2, Computational Efficiency), where we compare the runtime cost of SIHD with baseline methods such as Diffuser and HD. The results show that while SIHD does incur slightly higher training and inference time compared to HD, the overhead is modest and well-justified by the consistent performance gains. More importantly, SIHD retains a significant efficiency advantage over single-layer Diffuser models, reinforcing the practical feasibility and scalability of our approach.

To further address concerns about model capacity, we conducted an additional ablation on Single-task Maze2D in which we reduced the parameter count of each individual diffusion layer so that the total number of parameters across all layers matches that of the two-layer HD model. As summarized in the following table, even without increasing overall model capacity, increasing the number of hierarchical layers yields consistent and significant performance improvements. This demonstrates that the benefit of SIHD arises from its hierarchical structure rather than simply from added model size. We will include these discussions and results more explicitly in the final version of the paper.

$\bullet$ W4: In offline RL, it is indeed essential to avoid excessive extrapolation beyond the support of the dataset, as this can lead to unreliable value estimates. However, prior works [1, 2] have shown that controlled exploration—particularly toward underrepresented but plausible regions of the state-action space—can improve robustness to value estimation errors and enhance generalization, especially when guided by entropy-based regularization.

The challenge, as you noted, lies in balancing this exploration. Unconstrained deviation from the data distribution may introduce harmful out-of-distribution (OOD) behaviors. To address this, we introduce a structural entropy-based regularizer, which differs from conventional entropy maximization approaches in two key ways:

Structural Awareness: Rather than encouraging exploration uniformly across the entire state space, our method promotes diversity within the hierarchical community structure inferred from the offline data. This ensures that the exploration remains aligned with the intrinsic topological structure of the data, as captured by the encoding tree.

Controlled Diversity: The structural entropy metric formalized in Theorem 4.2 not only promotes generative diversity but also penalizes excessive deviation from the established hierarchical state structure. This provides a principled way to balance exploration and support preservation, effectively mitigating extrapolation risks.

Our ablation results further demonstrate that incorporating this regularizer improves performance compared to both no regularization and standard entropy-based exploration.

$\bullet$ W5: In our experimental evaluation, we use trajectory lengths up to 390 in Maze2D and 450 in AntMaze environments—substantially longer than typical short-horizon tasks—demonstrating SIHD’s capacity for long-range planning. To further investigate scalability, we conduct a detailed analysis in Appendix C.1 (Sensitivity Analysis), where we vary the number of diffusion layers from 2 to 6 across different scales of the AntMaze tasks.

The results indicate that increasing the number of hierarchy levels enables planning over longer time spans. However, we also observe that deeper hierarchies result in much shorter subtrajectory lengths per layer, which can degrade the generative performance of individual diffusion models. This trade-off highlights a key design consideration in balancing depth and subtrajectory complexity.

Moreover, in Appendix C.2 (Computational Efficiency), we compare the training and inference time across tasks of varying scales. Despite introducing additional diffusion layers, SIHD maintains computational efficiency comparable to the two-layer HD baseline, due to effective parallelism and reduced per-layer sequence lengths.

$\bullet$ W6: We have expanded our Qualitative Comparison (Figure 7) to illustrate exactly what each diffusion level learns on the Maze2D navigation task: low‑level diffuser generates fine-grained waypoints that navigate local obstacles, mid‑level diffuser proposes intermediate subgoals that bridge across small clusters of states—e.g., navigating between adjoining corridors, and high‑level diffuser identifies long‑range landmarks or critical “turning points” across the entire maze, allowing for flexible, multi‑scale planning.

By visualizing these subgoals side‑by‑side (and contrasting them with those from the two‑layer HDMI baseline), readers can immediately see how additional hierarchy levels yield increasingly abstract and spatially dispersed goals. Due to rebuttal submission constraints, we cannot include the updated figure here, but it will appear in full in the camera‑ready version after conference acceptance.

$\bullet$ W7: In the current version of the paper, we provide a complete pseudocode implementation of the inference procedure in Appendix A.5. To improve accessibility and clarity, we have now moved a high-level summary of this procedure into the main text and added explanatory commentary to better convey the planning mechanism.

At inference, our SIHD planner proceeds as follows (see Algorithm4): We first initialize each hierarchy’s subgoal buffers and the 1‑layer state–action sequence (lines 3 and 4). Then for each planning step, we (i) sample an initial noisy latent sequence from the diffuser prior (lines 6 and 7), (ii) invoke the subgoal proposer to revise the top‑level goal if its terminal criterion is satisfied (lines 8-10), and (iii) execute a reverse‑diffusion rollout across latent timesteps—at each diffuser step to sample the next latent sequence (lines 11-17). Finally, we decode and integrate the new state and action back into the 1‑layer sequence (lines 18 and 19), and after $H$ iterations output the resulting action trajectory (line 21).

References:

[1] Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.

[2] Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning. NeurIPS 2024.

Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.

$\bullet$ W1: To ensure fair comparisons, our initial experiments focused on benchmarks commonly used in hierarchical diffusion research, most of which are based on D4RL. However, to address the reviewer’s concern regarding generalizability, we conducted additional evaluations on the FrankaKitchen and NeoRL [1] benchmarks.

The results demonstrate that SIHD consistently achieves higher average returns across these diverse offline tasks. This not only reinforces the performance and generality of our framework but also empirically validates the feasibility of deriving hierarchical structures based on structural information from offline states beyond D4RL. These findings suggest that the state-space structure necessary for tree-structured partitioning can be reliably inferred from a variety of offline datasets, highlighting the scalability and broad applicability of our approach.

$\bullet$ W2: In our method, the accuracy of state community partitioning is ensured through the minimization of structural entropy, which serves as an unsupervised optimization objective for identifying coherent hierarchical communities. This approach has been widely adopted in hierarchical decision-making [2] and intrinsic reward design [3] within reinforcement learning.

To further address your concern, we quantitatively evaluated the quality of the partitioned communities on the Maze2D task using two standard unsupervised clustering metrics: Silhouette score (S) and Calinski-Harabasz index (CH). The results show that our structural entropy-based partitioning achieves $S>0.8$ and $CH>170$, indicating strong intra-community cohesion and inter-community separation—both hallmarks of high-quality clustering.

To assess the impact of community quality on performance, we designed an ablation variant, SIHD-FT, which replaces structural entropy-based partitioning with fixed-interval temporal segmentation. Experimental results show a clear drop in average return for SIHD-FT compared to the full SIHD model. This confirms that minimizing structural entropy not only improves partitioning accuracy but also plays a critical role in identifying key offline states and guiding the construction of effective hierarchical diffusion structures.

$\bullet$ W3: You are absolutely right that community partitioning introduces additional computational overhead, particularly in large-scale settings. To address this concern, we designed our system to perform state community partitioning entirely offline, storing the resulting structure as a dictionary. This avoids repeated computation during training and inference, ensuring that the hierarchical diffusion model remains practically deployable.

In Appendix C.2 (Computational Efficiency), we present a detailed comparison of training and inference time between SIHD and baselines such as Diffuser and HD across tasks of varying scale. Despite the inclusion of additional hierarchical layers, SIHD maintains comparable runtime efficiency to HD, thanks to parallelism and the reuse of precomputed structures.

$\bullet$ W4: While we provide strong empirical evidence demonstrating the benefits of the structural entropy regularizer, we also recognize the importance of theoretical support.

To that end, in addition to experimental validation, we include a formal theoretical analysis in Theorem 4.2, which establishes a lower bound formulation of Shannon entropy under a structural entropy framework. This result shows that our regularizer promotes exploration in a controlled manner: it encourages trajectory diversity while preserving the hierarchical community structure inferred from the offline data.

Unlike conventional entropy maximization strategies [4] that risk inducing unsafe out-of-distribution actions, our formulation explicitly constrains exploration within topologically meaningful regions of the state space. This provides a theoretical foundation for the regularizer’s ability to balance generalization with robustness, mitigating extrapolation error in offline settings.

References:

[1] NeoRL: A near real-world benchmark for offline reinforcement learning. NeurIPS 2022.

[2] Hierarchical State Abstraction Based on Structural Information Principles. IJCAI 2023

[3] Effective Exploration Based on the Structural Information Principles. NeurIPS 2024.

[4] Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning. NeurIPS 2024.

Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.

$\bullet$ W1: We have revised the paper to improve the clarity of writing and the presentation of algorithmic components. Specifically, we clarified the notation used throughout the paper: superscripts $h$ and $i$ denote the diffusion layer and temporal index within the same layer, respectively, while subscripts $g$ and $sa$ indicate subgoal sequences and state-action subtrajectories.

In response to the feedback on Figure 2, we have redrawn the figure to improve its readability. The three key modules are now visualized separately in distinct subfigures, allowing for a clearer depiction of the processing flow at each stage. Additionally, we expanded the introductory paragraph of Section 4 to briefly explain the roles of the three main modules:

Hierarchy Construction Module: captures the topological structure of offline states based on feature similarity and automatically uncovers hierarchical communities by optimizing structural entropy. These communities are then used to construct a flexible, multi-scale diffusion hierarchy.

Conditional Diffusion Module: inputs temporally segmented trajectories into corresponding layers of a shared diffusion model, integrating quantized rewards and structural signals to perform forward diffusion and reverse prediction at multiple time scales.

Regularized Exploration Module: leverages structural entropy as a measure of generative diversity and introduces a regularization loss to encourage the exploration of underrepresented offline states during training and inference.

These changes are intended to make the overall pipeline and technical intuition easier to follow.

$\bullet$ W2: As outlined in the Introduction and Related Work sections, although prior studies [1, 2] have proposed hierarchical diffusion models, they are typically restricted to a fixed two-level structure with uniform temporal segmentation. These methods lack the flexibility to adapt across varying time scales and essentially operate as two-layer diffusion models. In contrast, our work addresses a core open challenge in the field—designing a truly general and flexible hierarchical diffusion framework. To the best of our knowledge, this is the first method to realize a multi-level diffusion hierarchy that adapts to diverse temporal abstractions, making our contribution a substantive advancement rather than an incremental improvement.

$\bullet$ W3: To ensure fair comparison with prior work, our initial experimental setup focused on benchmarks commonly adopted in hierarchical diffusion studies [1, 2], the majority of which are based on D4RL. Nevertheless, to address the reviewer’s concern regarding generality, we conducted additional experiments on the FrankaKitchen and NeoRL [3] benchmarks.

In the NeoRL benchmark, we also included MB-PPO [3], a strong model-based baseline known for its performance in this setting. The experimental results demonstrate that SIHD consistently achieves higher average returns across different offline tasks. These findings further substantiate the effectiveness and generality of our proposed framework in diverse decision-making environments.

$\bullet$ Q1: In Sections 5.3 (Ablation Study) and Appendix C.1 (Sensitivity Analysis), we have already evaluated simplified two-layer diffusion variants of our model—SIHD-DH (with a fixed time horizon) and SIHD-2 (with structure entropy but limited to two levels). The results, summarized in the table below, show that both variants perform noticeably worse than the full SIHD model. This confirms the quantitative advantage of a deeper hierarchical diffusion structure over simpler two-layer architectures.

Additionally, in the Qualitative Comparison section, we extended Figure 7 to present visualizations of hierarchical subgoal generation on the Maze2D navigation task. Compared to HDMI, SIHD produces subgoals with more flexible and longer-range temporal transitions: low‑level diffuser generates fine-grained waypoints that navigate local obstacles, mid‑level diffuser proposes intermediate subgoals that bridge across small clusters of states—e.g., navigating between adjoining corridors, and high‑level diffuser identifies long‑range landmarks or critical ``turning points" across the entire maze, allowing for flexible, multi‑scale planning. This illustrates how the hierarchical design enables more globally coherent planning behavior.

Due to the limitations of the rebuttal format, we are unable to present the updated Figure 7 here, but we will include it in the camera-ready version after conference acceptance for improved clarity and illustration.

$\bullet$ Q2: The use of an encoding tree for hierarchical partitioning is grounded in the principle of structural information theory, which enables hierarchical decomposition of graph nodes without relying on any task-specific prior knowledge. This approach has been validated across various domains [4-6] for its effectiveness in capturing meaningful multi-level structure.

Importantly, the encoding tree supports a well-defined metric—structural entropy—which quantitatively evaluates the quality of the hierarchical partitioning. Unlike standard Shannon entropy, structural entropy promotes not only diversity among nodes but also preservation of the hierarchical community structure. This advantage is formally established in Theorem 4.2 of our paper.

By leveraging the encoding tree and structural entropy, our framework enables a flexible and general multi-level trajectory diffusion process. Furthermore, the integration of structural entropy as a regularizer promotes diverse exploration under the hierarchical structure, which enhances both model performance and generalization.

References:

[1] Hierarchical diffusion for offline decision making. ICML 2023.

[2] Simple hierarchical planning with diffusion. ICLR 2024.

[3] NeoRL: A near real-world benchmark for offline reinforcement learning. NeurIPS 2022.

[4] Robustness Evaluation of Graph-based News Detection Using Network Structural Information. SIGKDD 2025.

[5] Effective Exploration Based on the Structural Information Principles. NeurIPS 2024.

[6] Incremental measurement of structural entropy for dynamic graphs. AIJ 2024.