FracFace: Breaking The Visual Clues—Fractal-Based Privacy-Preserving Face Recognition

Dear Reviewer qSTt,

We sincerely thank you for your valuable comments and suggestions. We respond to your questions and comments below:

Response to Q1. On the Non-Invertibility of FFM

Thank you for this insightful question. The non-invertibility of the FFM is proved in [S1]. We conducted a quantitative analysis to support our claim, drawing from both entropy-based evidence and one-way permutation hardness.

The mapping defined by FFM is:

$

\mathcal{F}^{[k]}[i,j] = M_0[i,j] + \sum_{\ell=1}^{k} (b_\ell - 1) \cdot \beta_\ell, \quad \text{where } \beta_\ell = \prod_{s=1}^{\ell - 1} b_s,\ \beta_1 = 1.

$

This mapping grows rapidly with depth $k$, and the output is projected into a fixed index space of size denoted as $C$. Notably, each element of $\mathcal{F}^{[k]}$ increases proportionally with $\beta_k \in \mathcal{O}(3^{2k})$, implying that the range of $\mathcal{F}^{[k]}$ expands exponentially with $k$. To project the resulting indices into a fixed channel space, we define a modulo-based index mapping $\psi[i,j] = \mathcal{F}^{[k]}[i,j] \bmod C.$

However, due to the rapid growth of $\mathcal{F}^{[k]}[i,j]$, the modulo projection inevitably causes aliasing:

$

\exists\ (i_1,j_1) \neq (i_2,j_2) \quad \text{s.t.} \quad \psi[i_1,j_1] = \psi[i_2,j_2].

$

This proves that $\psi$ is not injective, and thus, the inverse $\psi^{-1}$ is not uniquely defined $\not\exists\ \psi^{-1} : \mathbb{Z}_C \to \mathbb{Z}^{n \times n}.$

This is not incidental, but structurally guaranteed due to the following:

1）The maximum index in $\mathcal{F}^{[k]}$ grows as $\max(\mathcal{F}^{[k]}) \in \mathcal{O}(3^{2k})$;

2）The output space of $\psi$ is fixed at size $C$;

3）Collisions are guaranteed. For example, when $C = 81$, $k = 2$, and $b_1 = b_2 = 3$, $\mathcal{F}^{[2]}$ yields values in \[1,162]\, projected into \[1,81]\ by mod.

The FFM transformation is non-invertible even under known parameters, due to its exponential growth in index space and subsequent modulo projection. This structural non-injectivity ensures that channel remapping cannot be deterministically reversed by an adversary, which underpins FFM's privacy-preserving design.

References

[S1] Double parameters fractal sorting matrix and its application in image encryption.

Response to Q2. Robustness Against Diffusion/GAN-based Reconstruction

Thank you for raising this important question. We extended our evaluation to stronger adversaries by including U-Net, StyleGAN, and Diffusion-based inversion models. As shown in Table R1, we benchmarked these attacks on SSIM, MSE, and IDS.

Table R1: Comparison of Reconstruction Attacks Against FracFace

While diffusion models can generate finer textures, they do not significantly outperform U-Net or StyleGAN in terms of privacy leakage under our setting. In fact, they slightly underperform in IDS and SSIM, and incur substantially higher computational cost (e.g., ~300s per image even with only 20 denoising steps).

Importantly, this experiment demonstrates FracFace’s resilience even against strong reconstruction adversaries with full knowledge of the fractal mapping. Despite that, FracFace still achieves low identity leakage and reconstruction quality, validating its effectiveness across diverse threat models.

Due to time limit, we used the SOTA pre-trained diffusion model []. It may be possible to train an end-to-end diffisuion or GAN inversion model that perfoms better than the pre-trained model. But we believe the improvement would not be significant, because FracFace fundementally removed detailed visual clues (see Fig. 3) that the reconstruction model relied on.

Response to Q3. Privacy–Utility Trade-off under Varying Fractal Layers and Pruning Ratios

We performed the ablation studies to investigate how fractal layers $k$ and pruning ratio affect the privacy–utility trade-off.

(1) Impact of Fractal Layers $k$

We varied the number of FFM iterations $k$ and evaluated both recognition accuracy and privacy robustness under StyleGAN-based reconstruction. As shown in Table R2, increasing $k$ leads to stronger privacy protection (LPIPS ↑, SSIM ↓), but accuracy begins to decline beyond $k=2$. We identify $k=2$ as the optimal point, offering a 20% LPIPS gain over $k=1$ with only a 0.02% drop in accuracy. Visualization results will be included in the final version, per policy.

Table R2: Effect of Fractal DepthLayers $k$ on Privacy–Utility Trade-off

(2) Impact of FBA Pruning Ratios

To evaluate sensitivity to pruning, we ranked DCT channels by global energy and retained the top-N at five pruning levels: 20%, 40%, 50%, 60%, and 80%. As shown in Table R3, stronger pruning improves privacy but degrades accuracy—especially beyond 50%. The 50% level provides the best trade-off: LPIPS improves by 35% over the 20% case, while accuracy remains high (99.69%).

Table R3: Effect of Pruning Ratios on Privacy–Utility Trade-off

These results confirm a clear privacy–utility trade-off. The setting with $k=2$, and 50% pruning achieves strong privacy protection while keeping recognition accuracy high. We will include these findings and full visualizations in the final version.

Response to Weakness 1

We thank the reviewer for raising this critical question regarding the novelty and security of our approach.

Firstly, prior works such as MinusFace and PartialFace have explored coefficient pruning in the frequency domain through static filtering. However, they do not address the deeper issue of frequency-domain sparsity, a structural vulnerability that makes frequency maps prone to reconfiguration and inversion. In contrast, FracFace explicitly targets this limitation through a novel fractal-based remapping mechanism, called FFM. As shown in our ablation results (Fig. 6 and Table 3), pruning alone is insufficient to disrupt reconstructive cues, and it is the FFM module that fundamentally improves privacy while preserving recognition performance.

Furthermore, FFM operates through recursive fractal expansion followed by a modulo projection, which inherently leads to non-injective index mappings—that is, multiple inputs may map to the same output. This structural aliasing ensures that even when the mapping function is fully known, it cannot be uniquely inverted. We provide a quantitative argument of this non-invertibility in Q1.

Finally, we operate under a strong white-box threat model, where the adversary is assumed to know the FFM mechanism. However, they do not know the specific instance of the random base Fractal Kernal $M_0$, and Fractal Lattice $L_0$, which varies across deployments and is not recoverable from outputs alone (see Q2).

We will include these analyses in the final version to clarify that FracFace innovatively applies fractal remapping to address the above challenge, moves beyond simple pruning via fractal remapping, and achieves provable non-invertibility even under strong threat models.

Response to Weakness 2

We address each concern as follows:

First, while LPIPS improvements may appear numerically small, we emphasize that privacy leakage is assessed through multiple complementary metrics. In addition to LPIPS, we report IDS, SSIM, and MSE, which together offer a fuller picture of reconstruction quality and identity preservation (see Table 2 in our paper). Next, we further evaluated FracFace under stronger adversaries, including U-Net, StyleGAN, and a diffusion-based inversion model. Detailed results and discussions are provided in Q2. Moreover, we have conducted ablation study on fractal depth and pruning ratio (see Q3). In terms of dataset coverage, FracFace have evaluated on 8 public datasets, including AgeDB, CFP-FP, IJB-B, and IJB-C, which introduce challenges such as pose variation, lighting, partial occlusion. In particular, CelebA, IJB-B,and IJB-C reflect unconstrained and non-frontal conditions. FracFace consistently outperforms baselines under these scenarios (Table 2), validating its robustness beyond frontal high-quality settings. Finally, on low-resolution scenarios, we note that degraded images often provide inherent privacy benefits by reducing recoverable visual details. This aligns with our core goal of obscuring identity while preserving utility.

Response to Limitations

Thank you for raising this important point. While a brief discussion of limitations is included in Appendix A.6 of the main paper, we agree that a more explicit and thorough treatment of the suggested issues would strengthen the work. We will incorporate these discussions in the final version. Due to the word limit of this response, we are unable to elaborate in detail here but would be happy to provide further clarification during the next review phase.

Looking forward to further discussion and feedback.

Dear Reviewer NRe7,

Thank you for your positive evaluation and encouraging remarks! We carefully investigated each question and weakness trying to address them to the best of our capability within the time frame. Our response are as following:

Response to Q1. Novelty

The paper presents incremental novelty: frequency-based PPFR approaches have long been explored in prior work. However, the authors do introduce new techniques to effectively reduce feature sparsity in the frequency domain, which is believed to be critical for defending against reconstruction attacks. The proposed modules are thoughtfully integrated into the frequency-based PPFR framework and demonstrate strong empirical performance.

We greatly appreciate your recognition of the technical contributions in our work, especially regarding the challenge of frequency domain sparsity. As you rightly pointed out, frequency-based PPFR pipelines have been widely explored. However, most existing approaches focus on filtering or masking coefficients, without explicitly tackling the structural sparsity in the frequency domain. This sparsity inherently permits reconfigurability and reversibility, posing a serious vulnerability under reconstruction attacks. In this work, we identify frequency domain sparsity as a key limitation that compromises the balance between privacy and usability. As shown in our ablation studies (Fig. 6 and Table 3 in our main paper), improvements in the frequency domain, for example through the FCR module, alone are limited in addressing the core challenge. This motivates the need for a novel approach. To this end, we innovatively leverage the inherent self-similarity and recursive locality of fractal structures, and design the FFM module to effectively mitigate frequency-domain sparsity. FFM enables FracFace to systematically overcome frequency sparsity and achieve robust privacy protection without sacrificing usability.

Response to Q2. Motivation & Performance

Among different options that could obfuscate reconstruction, what is the motivation for using fractal structures specifically? Could similar gains be achieved via other non-linear mappings or learned index transformations?

We thank the reviewer for this insightful question. The motivation for designing fractal structures is to defend against reconstruction attacks by disrupting the structure of frequency representations exploited by generative models. In developing this mechanism, we explored various alternatives, such as randomized nonlinear mappings and learned index transformations. We ultimately chose fractal structures because they offer a rare combination of effective reconstruction resistance and preservation of recognition performance.

Specifically, our choice of fractal-based frequency remapping is motivated by two intrinsic properties of fractals:

1. Local Recursion: Fractal mappings expand spatially through recursive integer rules, generating complex and multiscale index patterns that are not easily invertible. This recursive structure increases entropy in the transformed space and helps prevent reconstruction, even when the adversary has full white box access to the approach.

2. Self-Similarity: Despite structural complexity, fractals preserve local correlations across scales, which helps maintain the semantic consistency of identity-related signals. This allows recognition models to extract usable embeddings, preserving utility even under aggressive obfuscation.

These two properties complement each other and together enable us to tackle a core problem in PPFR: balancing privacy and usability. As shown in our results (Table 1 and 2), FracFace significantly boosts resistance to reconstruction attacks (e.g., 60% LPIPS increase), while keeping recognition accuracy almost unchanged (e.g., only 0.04% drop on LFW), highlighting its effectiveness in achieving this balance.

We explored several alternative transformations, including:

• Random or pseudo-random channel permutations, which are easy to reverse if known and often degrade utility due to lack of structural regularity;
• Learned index transformations, which introduce additional parameters and may suffer from overfitting or instability under unseen conditions;
• Frequency-space scrambling based on fixed priors, which often fail to generalize across identities and domains.

While these alternatives had some potential, they failed to meet two key requirements of our approach: structural non-invertibility and semantic preservation. In contrast, fractal mappings are deterministic yet chaotic, simple yet expressive. This unique combination allows us to build a defense that is both mathematically analyzable and practically effective. We will add this analysis.

Response to Q3. On the Sensitivity to FCR Pruning Strategy

How sensitive is FracFace to the choice of pruning strategy in FCR? Have the authors considered adaptively learning the channel selection or making it adaptive to image content?

Thank you for your thoughtful question. In our work, we carefully analyzed the sensitivity of pruning strategies. Our choice was guided by both empirical results and prior studies, which show that overly aggressive or overly conservative pruning often leads to unstable or suboptimal performance. To understand this behavior in greater detail, we focused our analysis on the 40%–60% pruning range, where the transition became particularly evident. As shown in Table R1, we found that retaining approximately 50% of the DCT channels (i.e., 81 out of 192) provided the best trade-off: it significantly improved visual obfuscation (LPIPS increased from 0.3184 to 0.6839) while maintaining high recognition accuracy (99.69%).

Table R1: Effect of Refining Ratio on Privacy–Utility Trade-off

In fact, the process of selecting a pruning strategy can be seen as a form of adaptive learning process that evaluates how channel selection affects privacy and utility.

In practical applications, adapting the pruning strategy to each image introduces non-negligible computational overhead and deployment cost—particularly since FracFace operates on high-resolution images (112×112). For inputs with consistent pixel dimensions, our experiments show that a fixed pruning strategy offers greater efficiency and stability. Therefore, we adopt a fixed pruning approach and recommend cropping images to a consistent resolution when using FracFace.

We appreciate your suggestion and will add the above in the final verison.

Response to W1. Motivation & Ablation

The motivation behind fractal kernel construction could be more intuitive; some components are overly complex without clear ablation.

Thank you for the valuable comment. We would like to clarity the intuition behind the fractal kernel and will include it in the final version. As detailed in our response to Q2, the design is motivated by optimizing the trade-off of between privacy and utility, leveraging the unique properties of fractals. We have also examined ablation effects related to FCR pruning sensitivity in Q3. Moreover, we conducted an additional ablation study, varying the number of fractal iterations $k$, to assess both recognition accuracy and privacy robustness under a StyleGAN-based reconstruction attack. As summarized in Table R2, increasing $k$ consistently enhances privacy (LPIPS ↑ from 0.53 to 0.84; SSIM ↓ from 0.52 to 0.26), indicating stronger visual obfuscation. However, recognition accuracy starts to decline beyond $k=2$. We find $k=2$ strikes the best balance, achieving a 20% LPIPS improvement over $k=1$ with only a negligible 0.02% accuracy drop. We will add these findings to the final version to support the discussion.

Table R2: Impact of Fractal Depth $k$ on Privacy–Utility Trade-off

Response to W2. The Reliance on Heuristic Pruning Rules in Frequency Channel Refining Seems to Lack Adaptivity

The reliance on heuristic pruning rules in frequency channel refining seems to lack adaptivity.

Thank you for the insightful comment. We have discussed this issue in Q3, where we evaluated the trade-off between pruning strength and performance, and highlighted that a fixed strategy achieves stable results across typical input resolutions (e.g., 112×112). While content-aware refining could offer finer control, it may also introduce additional computational cost. We will clarify this rationale in the final version and include adaptive refinement as a potential direction for future work.

We hope we have addressed your questions. Please let us know your further concerns during discussion.

Dear Reviewer NRe7:

We appreciate your continued engagement with our work.

If there are any aspects that you feel could benefit from further clarification, we would be happy to address them as we finalize the paper. We remain committed to improving the clarity and overall quality of the submission.

We truly value your insights and would greatly appreciate any further suggestions you may have. We will be staying online and responsive in the coming days, should there be anything we can clarify or improve further.

I appreciate the authors' response. After considering their rebuttal and the feedback from my fellow reviewers, I have decided to maintain my original score.

Dear Reviewer qzRH,

Thank you for your positive evaluation and encouraging remarks! We carefully investigated each question and weakness trying to address them to the best of our capability within the time frame. Our response are as following:

Response to Q1. Ablation on Fractal Depth $k$

Please include a systematic study of how varying the number of FFM iterations (k) affects both privacy and recognition accuracy.

Thank you for this constructive suggestion. We conducted the requested ablation study by varying the number of fractal iterations $k$, evaluating both recognition accuracy and privacy robustness under the StyleGAN-based reconstruction attack.

The results are summarized in Table R1 below. As $k$ increases, privacy improves consistently—LPIPS increases from 0.53 to 0.84, and SSIM drops from 0.52 to 0.26—indicating stronger obfuscation. However, recognition accuracy begins to degrade beyond $k=2$. Notably, $k=2$ emerges as the optimal trade-off, offering a 20% LPIPS gain over $k=1$ with a minimal 0.02% drop in accuracy. The results on StyleGAN reconstruction images cannot be upload in the response. We will include them in the final version.

Table R1: Impact of Fractal Depth $k$ on Privacy–Utility Trade-off

Response to Q2. Sensitivity to FBA Pruning Strength

“Can you quantify the impact of different low-frequency attenuation levels on privacy vs. utility?”

Thank you for the insightful suggestion. To assess the sensitivity of our system to FBA pruning strength, we conducted additional experiments varying the refining ratio in the FCR module using an energy-guided channel pruning strategy. Specifically, we ranked DCT channels by global energy and retained the top-N channels corresponding to five pruning levels: 20%, 40%, 50%, 60%, and 80% (inspired by FaceMAE). We evaluated the effect of each level on feature sparsity, privacy robustness (SSIM / LPIPS), and recognition accuracy. The results are presented in Table R2 below.

Table R2: Effect of FBA Pruning Strength on Privacy–Utility Trade-off

We observe a clear privacy–utility trade-off curve: as refining becomes stronger (i.e., more channels are removed), privacy consistently improves (e.g., LPIPS: 0.32 → 0.86), but recognition utility drops, especially beyond 50%. Notably, 50% pruning yields the best balance, increasing LPIPS by 35% over the 20% case while maintaining near-perfect recognition accuracy (99.69%). Beyond that point, privacy stays about the same, while accuracy keeps going down (e.g., 60% → 89.26%, 80% → 87.24%). These results highlight that moderate refining (40–50%) forms an optimal operating range, effectively improving privacy with minimal impact on utility. This justifies our default choice of 50% refinement in the main paper. We will include this ablation and its discussion in the final version. The figure on privacy-utility curve cannot be upload in the response. We will include them in the final version. We thank the reviewer for encouraging this more comprehensive analysis.

Response to Q3. Definition and Computation of Protection (%)

“Specifically, state what quantities appear in the numerator and denominator, and include a short numerical example.”

Thank you for your insightful question. We agree that the computation of Protection (%) warrants a precise definition and will add it to the final version along with a numerical example.

Definition: Protection (%) represents the share of frequency-domain channels that are either (i) filtered out by FCR or (ii) structurally disrupted by FFM, as a percentage of the total number of channels.

Protection (%) = $\frac{|P_{mask}|+|P_{rempa}|}{P_{total}}$

Where:

• $P_{\text{total}}$: total number of DCT frequency channels (e.g., 192 for 12×16 DCT).
• $P_{\text{mask}}$: number of low-energy channels pruned by FCR.
• $P_{\text{remap}}$: number of remaining channels remapped by FFM.

Here is a numerical example illustrating the calculation of Protection (%):

Table R3: Example calculation

This unified formulation allows for consistent, interpretable comparison of visual disruption levels across methods, regardless of the specific masking or remapping mechanism.

Response to Q4. Pseudocode

The box labeled “Refined Frequency Channel” shows outputs S(1) and S(2) but gives no indication how they are computed or split.

Thank you for the helpful suggestion. We acknowledge the specific calculation process of S(1) and S(2) lacked clarity and appreciate the opportunity to improve our explanation. To construct the two refined frequency channel groups S(1) and S(2), we apply a index-like traversal over a frequency index grid. The pseudocode is as follows:

# Step 1: Create frequency index grid
F = create_matrix(M=12, N=12)

# Step 2: S-shaped traversal
freq_list = []
for i in range(M):
    row = F[i]
    if i % 2 == 0:
        freq_list.extend(row)         # left-to-right
    else:
        freq_list.extend(row[::-1])   # right-to-left

# Step 3: Split into two groups
S1 = freq_list[0:80]
S2 = freq_list[80:161]

The indexing method is illustrated in Fig. 8. We will include the pseudocode and a sub-diagram showing how S(1) and S(2) are generated to improve clarity in the final version.

Response to Q5. The Sensitivity Analysis of Different Frequency Bands

Which specific frequency bands contribute most to privacy leakage or recognition performance?

Thank you for the insightful suggestion. Following your advice, we conducted a per-band frequency sensitivity analysis to investigate which frequency ranges contribute most to privacy leakage and recognition performance. As shown in Table R4, we conducted both “retain-only” and “mask-only” experiments on three grouped frequency bands (Low, Mid, High):

• Low-frequency channels are strongly correlated with visual appearance and structure. Masking them significantly improves privacy (↓SSIM, ↑LPIPS) without hurting recognition (99.41% accuracy), confirming their dominant role in visual leakage.
• Mid-frequency channels are most critical for identity representation. Retaining only mid-bands achieves 97.69% accuracy, while masking them causes a notable drop to 92.15%, indicating their contribution to discriminative identity cues.
• High-frequency channels provide limited standalone utility but help encode textures. Masking them has little impact on accuracy but slightly reduces privacy performance. (Clarity: We just provide the result data, as shown in Table R3. And we are unable to provide the visualization result of heatmap per the rebuttal policy, but we will put the result in the final version)

Table R4 The Sensitivity Analysis of Different Frequency Bands

Response to W1. Ablation on Fractal Depth k

Please see our response to Q1 for detailed analysis.

Response to W2. Sensitivity to FBA Refining Strength

Please see our response to Q2 for detailed analysis.

Response to W3. The Sensitivity Analysis of Different Frequency Bands

Please see our response to Q5 for detailed analysis.

Response to W4. Definition and Computation of Protection (%)

Please see our response to Q3 for detailed analysis.

Response to W5. Clarifying “P = P₁ ∪ P₂ ∪ P₃” and the generation of S(1)/S(2)

Thank you for pointing out this important issue. We confirm that our implementation follows the formula $P = P₁ ∪ P₂ ∪ P₃$ as stated in line 159. After converting the input image to the frequency domain, we retain a three-layer structure, where each $P_{i}$ corresponds to a group of channels ordered by energy. Specifically, $P$₁ captures dominant identity-relevant signals, while $P₂$ and $P₃$ reflect more fine-grained appearance traits such as illumination and texture. Our FCR module applies layer-wise frequency refining to remove appearance-related components while preserving discriminative features for recognition. Regarding the separation of the refined channels into $S(1)$ and $S(2)$, we apply an S-shaped scanning pattern across the frequency grid and split the resulting index sequence evenly into two subsets. This design ensures spatial and frequency diversity across both groups (see our response to Q4 for more detail). We appreciate this helpful suggestion and will revise Fig. 1 in the final version to clearly reflect both the layered pruning structure and the generation of $S(1)$ and $S(2)$.

We hope we have addressed your questions. Please let us know your further concerns during discussion.

Dear Reviewer sFx1,

Thank you for your valuable comments and suggestions. Our detailed point-by-point responses are provided below.

Response to Q1. On the Privacy–Utility Trade-off

We would like to clarify the concern as follows. In practice, minor reductions in recognition accuracy are acceptable, since modern face recognition systems adjust thresholds based on deployment context, with stricter settings for border control and more lenient ones for surveillance.

This trade-off is a well-known challenge in the literature, where enhanced protection often comes at the cost of utility. For example, FaceObfuscator [S1] explicitly discusses the usability–privacy trade-off in real-world deployments, while PPFR-FD [S2] demonstrates that only when removing over 94% of frequency channels would significantly reduce recognition accuracy. Our method retains 50% of these channels, offering both structural protection and utility, as shown in Table R1.

Table R1: Minimal Utility Drop of FracFace Compared to ArcFace

References

[S1] FaceObfuscator. USENIX Security 2024.

[S2] Privacy-Preserving Face Recognition in the Frequency Domain. AAAI 2022.

Response to Q2. Compatibility with HE and FL

The suggestion is interesting! As a modular model-agnostic preprocessing layer operating at the feature level, FracFace is inherently well-suited for combination with HE and FL, offering complementary protection across different stages of the pipeline. Specifically, FracFace enhances input-level visual privacy by obfuscating frequency and spatial cues prior to downstream processing.

Response to Q3. Can residual visual cues reveal age, gender, or ethnicity?

Thank you for this thoughtful question. FracFace is specifically designed to suppress visual cues that contribute to face reconstruction. While it is possible that certain soft-biometric attributes may partially persist after transformation, these signals alone are generally insufficient for reliable identity inference by an adversary.

To examine this concern, we conducted a user study on 20 FracFace-processed celebrity images. Participants (54 responses) were asked if they recognized the person and which cues they used. As shown in Table R1, most relied on features like face shape and hairstyle, not soft-biometrics. Specifically:

Table R1: User Reliance on Soft-Biometric Attributes for Identity Inference

Over 76% of participants reported relying on guesswork or vague impressions, citing a lack of clear visual cues for recognition.

Response to W1. Novelty

Thank you for the comment.

While frequency-domain processing is common in PPFR, we address a novel challenge: the sparsity-driven reconstructibility of DCT features. Our FCR module prunes channels based on semantic relevance rather than fixed frequency bands, reducing redundancy. However, as shown in Fig. 6 and Table 3, this alone is insufficient. To go beyond frequency space, we propose a novel fractal-based mechanism that remaps DCT features into a recursively constructed, non-invertible fractal space. To our knowledge, this is the first use of fractal geometry to disrupt structural coherence in the frequency domain (see Appendix A.2).

Response to W2. On the Efficiency of FFM, Challenging Scenarios

We thank the reviewer for raising this valuable point.

We emphasie that this work aims to address the core challenge of balancing privacy and utility in PPFR, which requires to disrupt spatial regularities (resisting reconstruction), while preserving identity-related features to support recognition. Therefore, we evaluated the effeciency with regard to face recoginition accuracy and privacy preservation (measured by SSIM, IDS, MSE, and LPIPS). Compared with 6 SOTA ways to obfuscate feature (Table 1 and 2), our expeirements show that FFM is the most effecient for this specific purpose. Some of existing works e.g., FaceObfuscator and Minusface, can obfuscate features, but the results can be easily reverted, and thus unsuitble for this work.

Regarding computational effeciency, we have provided the FFM complexity calculation in Appendix A.5 Training Details.

Recognition accuracy drop remains negligible (0.04%-2% in our experiments) under practical deployment thresholds, and FracFace performs robustly across challenging conditions. Datasets like IJB-B, IJB-C, and CelebA feature real-world variations in pose, lighting, and resolution, yet FracFace consistently preserves high recognition accuracy while enhancing privacy protection.

Response to W3. Residual Privacy Risks and Visual Clues

Thank you for the insightful question. We consider a threat model where an adversary aims to reconstruct and identify faces from obfuscated images, assuming knowledge of FracFace parameters (e.g., fractal depth, expansion schedule). While visual cues cannot be fully removed due to recognition needs, FracFace greatly reduces them compared to prior work. Soft-biometric traits like face shape may remain but are beyond this work’s primary scope. Crucially, our transformation is non-invertible and does not introduce new privacy risks.

In detail, two components, the initial fractal kernel $M_0 \in \mathbb{Z}^{n \times n}$, and the index lattice $L_0 \in \mathbb{Z}^{n \times n}$, are secret and randomized per deployment, ensuring unique, secure instances. In addition, the fractal index mapping is defined recursively as:

This is then projected modulo the channel dimension $C$:

Because $\mathcal{F}^{[k]}$ grows exponentially with $k$, many different positions collide to the same index under $\psi$. That is,

This shows that $\psi$ is non-injective by design, and no inverse mapping $\psi^{-1}: \mathbb{Z}_C \rightarrow \mathbb{Z}^{n \times n}$ exists. Even if the adversary learns the full mapping rule $\psi$, they cannot resolve the original positions or channel correspondences without knowing the randomized $M_0$ and $L_0$.

These design choices ensure even subtle visual cues are unrecoverable under strong white-box attacks. The non-invertibility of $\psi$ and randomness from $M_0$ and $L_0$ jointly establish a robust privacy boundary, making FracFace highly resistant to reconstruction.

Response to W4. Adversarial Threat Model

We thank the reviewer for the concern. While our experiments use generative reconstruction attacks (e.g., StyleGAN inversion, U-Net), these simulate worst-case white-box threats (see W3), not just synthetic scenarios. Our assumed attacker has full access to model details and data priors—far beyond typical real-world adversaries. Yet, FracFace remains highly resistant to reconstruction, achieving:

• Over 60% drop in SSIM and 80% increase in LPIPS compared to unprotected images;
• Effective removal of visual cues even when all system components (excluding secret seeds) are known.

This threat model subsumes real-world conditions like lighting, pose, and domain shifts. Since FracFace withstands such strong attacks, its robustness extends to more typical, weaker adversaries.

Response to L1. Attack Diversity

We additional evaluated against a recent Diffusion-based method.

Table R1: Comparison of Reconstruction Attacks Against FracFace

Response to L2. Low-Resolution/ Noisy Inputs

See W2.

Response to L3. Identity Inference/Psychological Privacy Risks

To assess broader privacy, we conducted a user study (see Q3). 76% wrongly guessed, indicating weak visual cues. This suggests FracFace effectively limits identity inference and psychological privacy risks.

Looking forward to further discussion and feedback.