KaRF: Weakly-Supervised Kolmogorov-Arnold Networks-based Radiance Fields for Local Color Editing

Thank you for your thoughtful comments and constructive feedback. We conduct extensive experiments to address your feedback (including excellent results with 1/3 input masks, more comprehensive ablation studies, and 360° videos of mip360). However, due to the official rule against adding images in the rebuttal, we provide ample quantitative results and describe some of the visual outcomes. We promise that if the paper is accepted, we will include more visual experiments.

Q1: Supervision Level and Use of SAM? How many masks for training? How to prompt SAM?

Response: We define weak supervision by two key properties: the use of sparse and imperfect labels. Our method requires only 1-3 coarse masks per scene (one for forward-facing like LLFF, three for 360° scenes like Mip360). These masks are automatically generated by prompting SAM with points and are used directly without any manual refinement.

Though we may use more than one mask, our supervision is more efficient than prior work. Unlike IReNe, which requires precise manual maps and new reference images for multiple edits, KaRF uses coarse, auto-generated masks and modifies a single learned palette. Furthermore, unlike ICENeRF (often requires at least two masks: foreground and background), whose foreground mask requirement scales with the number of colors being edited, KaRF edits multiple objects within the same sparse set of masks.

We apologize for the confusion caused by Figure 4. The dense-mask setup shown was used exclusively to adhere to the SPIn-NeRF benchmark protocol for a fair comparison and does not represent our method's standard workflow.

Q2: Viewdep Inconsistency? the input-output relationship?

Response: We apologize for the lack of clarity in Figure 3. The confusion stems from the legend: the view direction input is used only for local recoloring, not segmentation. Its enclosure in the red dashed box visually signifies its exclusive role in the recoloring process.

To directly clarify the input-output relationships for each stage:

1. Segmentation Stage (View-Invariant): Inputs: 3D position + prior features. Output: consistent segmentation maps. This stage learns the intrinsic geometry and semantic class of the scene, which is independent of the viewing angle.

2. Local Recoloring Stage (View-Dependent): Inputs: 3D position + view direction + prior features. Output: consistent weight maps and a palette. Composites colors by location and viewing angle to reconstruct view-dependent effects (gloss, shading, reflections).

Q3: Clarification on the L_palette and view-dependent/independent effects?

Response: Unlike PaletteNeRF/IReNe, KaRF's view-dependent weights W model all color effects (diffuse, specular) with a view-invariant palette P. This is enabled by LoCoPalette (including black/white bases), supporting our two-phase training:

1. First 2k Iterations: Lweight provides a coarse, view-invariant guide to color distribution. Simultaneously, Lpalette (with a fixed palette P) acts as a strong view-dependent loss, enabling W to learn all necessary details for ground truth approximation. This includes utilizing white base for specular highlights and black base for shadows, with all diffuse, highlight, and shadow weights learned within W.

2. Last 1k Iterations: Palette P becomes learnable. The L_palette now also adaptively fine-tunes the base colors in P to achieve the best possible color fidelity for the reconstructed image.

You have made an excellent point regarding the naming of L_palette. To better reflect its dual role, we will rename it to Palette-Adaptive RGB Reconstruction Loss.

To visualize and quantify effect separation, our visualizations confirm that diffuse weights stay stable when the viewing angle changes for a fixed 3D point. Conversely, white and black weights vary significantly with view direction, precisely how view-dependent highlights and shadows are formed. An ablation study in Tab 1, with image contrast amplified 4x to highlight specular effects, quantifies this impact, demonstrating significant improvement from view-dependent weights in capturing complex lighting compared to a view-invariant model.

Tab 1: Weight W.

Q4: Strategy for background or other classes?

Response: KaRF is a modular framework that exclusively edits user-specified foreground objects, leaving the background unmodified. During training, background regions learn to select only a black palette base, effectively isolating them from the recoloring process.

For scenes with multiple classes, a single, shared palette and a unified weight field W are used. The distinct color of each object is then controlled by its specific learned weights within this unified weight field.

Q5-: Single input mask evidence?

Response: For LLFF, we require only a single mask to achieve high-quality segmentation. For Blender and Mip360, we use three masks from minimally-overlapping viewpoints.

We performed extensive experiments to validate this capability. The results of Tab 2 show that the metrics achieved with this sparse-mask supervision are nearly identical to the results from using a much larger set of masks. Qualitatively, the generated segmentations and the final local edits are visually indistinguishable from the results obtained with denser supervision. This demonstrates our model's strong generalization ability, allowing it to learn a complete and accurate segmentation field from extremely sparse signals.

Tab 2: Sparse masks input (IoU↑ Acc↑).

Q6-: Impact of GRBFKAN, L_palette, residual adaptive gating KAN and time comparison?

Response: We sincerely apologize that our initial experiments were not sufficient to fully demonstrate the generality and superiority of our design choices.

1. Impact of GRBFKAN: To demonstrate that our performance gains stem from the KAN architecture itself and not simply the supervision level, we conducted a broad comparison of MLP, B-Spline KAN, and GRBFKAN across six scenes from three datasets. The results of Tab 3 show that GRBFKAN consistently yields the best PSNR and SSIM, confirming its superior ability to model high-frequency color details compared to the smoother B-splines and global MLP. The results of Tab 4 confirm that GRBFKAN achieves the best average IoU and Acc, demonstrating its enhanced ability to learn complex geometric and semantic boundaries. Qualitatively, GRBF KAN also produces sharper segmentation boundaries and more accurate color compositions that are faithful to the original scenes.

Tab 3: Impact of the GRBFKAN (PSNR↑ SSIM↑).

Tab 4: Impact of the GRBFKAN (IoU↑ Acc↑).

2. Impact of L_palette: The results from three new scenes confirm that including our L_palette loss consistently and significantly improves reconstruction quality (PSNR/SSIM) and prevents color deviation across all dataset types:

Tab 4: Impact of L_palette (PSNR↑ SSIM↑)

3. Extended Time Comparison: We provide runtimes for three scenes from different datasets, demonstrating the stability and generalizability of our method's efficiency:

Tab 5: Processing Times

4. Impact of Residual Adaptive Gating KAN: We performed new ablations on three scenes for both segmentation and recoloring. The results of Tab 6 and 7 show our full design is consistently optimal. While its segmentation score is similar to w/o Res, it is significantly better than other variants, and as shown in Supp. Fig. 7, it converges much faster.

Tab 6: Impact of structure (IoU↑ Acc↑)

Tab 7: Impact of structure (PSNR↑ SSIM↑)

Q7-: Mip360 evidence?

Response: We apologize for the oversight in our initial submission's video presentation. Our method's superior multi-view consistency is already demonstrated quantitatively in Table 3 of the supplementary material, and we have now rendered a full 360° video for the Mip-360 dataset as further visual proof, which we are unable to upload here.

Q8: Ethical considerations?

Response: First, we use only public, non-sensitive benchmark datasets. Second, our supplementary material (Sec. 10) discusses both the positive societal impacts and the potential for misuse.

Ok thank you now it's clear, it was my bad, I was thinking for the palette and the weights as one single thing, but they are separed, and the W is the one view-dep.

Is there any reason why not all the scenes of mip360 were taken a subjects for the experiments?

I would have valued more experiments only done on mip360, than this division of some LLFF, some Synthetic, and some mip360. For example the garden scene, with the table, would have been a nice test to check the consistency of the view-dependent effects learned on the table

Thank you for the questions you have raised. We will address each of them in turn below.

Q1: The given implementation details are not sufficient to reproduce the method without the release of the codebase. The author also did not specify whether they plan to release the code.

Response: We sincerely apologize that the details in our paper and supplementary material were not sufficient to fully address your concerns about reproducibility.

We endeavored to provide the key information required to reproduce our method. To be as clear as possible, we have detailed the following in the Implementation Details subsection of section 5: The optimizer used for all stages. The learning rates and batch sizes for the different training stages. The specific number of iterations for each training stage. The weight hyperparameters for all loss functions. Furthermore, in section 4, we provided a detailed network architecture diagram (Figure 3), which includes our core residual adaptive gating KAN structure. We also supplied the precise mathematical definitions for all loss functions and the equations for segmentation and local recoloring. We believe this information provides a solid foundation for reimplementation.

Most importantly, we want to state our open-source commitment unequivocally. As mentioned in our response to the NeurIPS checklist 5, we will make our work fully accessible. Upon acceptance of the paper, we will immediately release the complete source code, all pre-trained models, and the pre-processed data to ensure full reproducibility.

Q2: The authors did not specify how many datasets were used for experiments and the change in implementation for different types of NeRF scenes.

Response: We apologize for any lack of clarity in our paper. Regarding the datasets used, we would like to clarify that we specified the three distinct datasets in the datasets subsection of section 5. The datasets are: NeRF-Synthetic (Blender), Mip-NeRF 360, and LLFF. For our qualitative evaluation (Figure 5), we selected one representative scene from each of these datasets. Similarly, our quantitative evaluation (Table 2) was conducted on a total of nine scenes drawn from across these same three datasets to ensure a comprehensive assessment.

Regarding your second point on implementation changes for different NeRF scene types, we thank you for raising this. We provided a single, unified set of implementation details because our proposed KaRF framework demonstrates excellent robustness and generalization. The same set of hyperparameters and training configurations was applied across all three distinct dataset types without needing any scene-specific or dataset-specific tuning. For the NeRF-Synthetic, LLFF, and Mip-NeRF 360 datasets, the learning rates, iteration counts, and loss weights listed in the implementation details subsection were used consistently.

Q3: The runtime comparison is not extensive and significant enough through simply providing "roughly 10 min" without specifying the type of scenes.

Response: We apologize if our description of the runtime was not sufficiently detailed. We chose the Fern scene for our exemplary comparison, a practice aligned with other methods such as LAENeRF, to provide a fair time comparison on a common and publicly available benchmark.

More importantly, we wish to clarify that the runtime for our method is highly stable and consistent across different scenes and dataset types. This is because the total processing time is primarily determined by the fixed number of training iterations and the number of rays processed per batch, rather than the visual complexity of the content of the scene. In our experimental setup, the training iterations and other settings are kept constant for all datasets, including NeRF-Synthetic, LLFF, and Mip-NeRF 360. As a result, the time required for local color editing remains nearly identical regardless of the scene. To provide concrete evidence, Tab 1 shows the processing times for several representative scenes from different datasets:

Tab 1: Processing Times.

Q4: How does the model handle the view-dependent effects such as specularity or transparency in editing? How would the model behave for scenes with complex lighting?

Response: While individual edges in a KAN learn univariate functions, we would like to clarify that this does not preclude the network as a whole from learning complex, cross-dimensional dependencies, such as those between 3D locations and view directions. Our model effectively handles view-dependent effects through the following mechanisms:

1. Explicit View Direction Input: In the local recoloring stage, which is responsible for effects like specularity and transparency, we provide the view direction as an explicit input to the network, following the successful practice of the original NeRF. This is formally shown in Equation (10) of our paper.

2. Expressive Power of the Composed Network: Although each edge activation is a univariate function, the full network, through its multi-layer composition, is fully capable of learning complex mappings from high-dimensional inputs to high-dimensional outputs. Our proposed residual adaptive gating KAN structure is designed to learn these non-linear dependencies between position and view direction effectively.

3. Qualitative Evidence: Our experimental results demonstrate the effectiveness of this design. In the Horns scene in Figure 5 of the main paper, our method successfully preserves the surface gloss and tonal transitions after changing the color of the horns. This gloss is a classic view-dependent effect, and its preservation proves that our model accurately learned and reproduced this dependency, a task where other methods like LAENeRF show visible artifacts.

Regarding the behavior of our model in scenes with complex lighting: The Mip-NeRF 360 dataset is representative of the scenes you describe, containing unbounded, real-world environments with complex lighting, shadows, and reflections. Our method achieves excellent editing results across multiple scenes from this challenging dataset, such as the Kitchen scene (main paper Figure 5) and the Bonsai scene (supplementary material Figure 11). In these scenarios, KaRF not only accurately replaces the color of the target object but also naturally integrates the new color into the existing lighting environment of the scene, correctly rendering the resulting highlights, shading, and shadows on the surface of the object.

Q5: The reason to adapt KAN for the specific task of localized NeRF recoloring is not well conveyed through the writing of this paper, and the authors have not shown enough results to demonstrate superiority of the method.

Response: We sincerely apologize that our initial manuscript did not sufficiently convey the core motivations for choosing the KAN architecture. We appreciate you raising these critical points, and we hope to offer a much clearer justification here. Our primary reasons for adapting KAN for the specific task of localized NeRF recoloring are as follows:

1. Superior Non-linear Expressiveness: Local color editing is an inherently complex function-fitting problem. KANs possess stronger function approximation capabilities and greater expressive power than traditional MLPs, making them better suited for this task.

2. Precise Modeling of High-Frequency Details: We further modified the KAN architecture to use GRBF instead of B-splines. This decision was driven by the fact that local editing often involves high-frequency details like specular highlights and sharp edges. Compared to the smoothing properties of B-splines, GRBFs can more flexibly adapt to and reconstruct these rapidly changing local features.

3. Higher Training Efficiency and Stability: Figure 7 in our supplementary material provides empirical evidence for the advantages of GRBF KAN. In both the segmentation and local recoloring stages, the training loss for GRBF KAN is consistently lower and converges faster than that of both MLP and B-Spline KAN architectures.

To address your valid concern about the lack of sufficient results, we have conducted a broad set of new quantitative experiments to systematically demonstrate the superiority of our GRBF KAN architecture. We selected two representative scenes from each of our three datasets and performed a comprehensive comparison between GRBF KAN, B-Spline KAN, and MLP.

1. Local Recoloring Task: We evaluate the reconstruction quality after palette synthesis. The results in Tab 2 show that GRBFKAN outperforms B-Spline KAN and MLP across all tested scenes in terms of PSNR and SSIM.

Tab 2: Impact of the GRBFKAN (PSNR↑ SSIM↑).

2. Segmentation Task: As shown in Tab 3, GRBFKAN consistently outperforms both B-Spline KAN and MLP in average IoU and Acc. This provides strong empirical support for our claim that GRBFKAN possesses a superior capacity for learning scene geometry and precise semantic boundaries.

Tab 3: Impact of the GRBFKAN (IoU↑ Acc↑).

Thank you for your detailed review of our work. We have carefully considered your comments and offer our responses below.

Q1: Limited Analysis of KAN Modifications.

Response: We thank the reviewer for highlighting the importance of thoroughly analyzing our GRBF-based modification to KAN. We agree that this is a central contribution and have conducted additional experiments to provide a more rigorous evaluation.

Expanded Evaluation: We systematically compared GRBFKAN with both B-Spline KAN and MLP across six representative scenes (two per dataset) and two key tasks: local recoloring and segmentation.

1. Local Recoloring: As shown in Tab 1, GRBFKAN consistently achieves the best PSNR and SSIM across all scenes. This supports our claim that GRBFs, with their flexible locality and sharper kernel responses, are better at capturing the high-frequency details present in colors and textures than smoother B-splines or global MLPs. Additionally, GRBFKAN exhibits faster and more stable convergence (see supplementary material Figure 7).

Tab 1: Impact of the GRBFKAN (PSNR↑ SSIM↑).

2. Segmentation: As shown in Tab 2, GRBFKAN consistently outperforms both B-Spline KAN and MLP in average IoU and Acc. This suggests its improved ability to learn geometric and semantic boundaries, which is critical for high-quality editing. Furthermore, the loss convergence plots (see supplementary material Figure 7) reinforce this advantage, showing that GRBFKAN not only converges faster but also achieves a significantly lower final loss.

Tab 2: Impact of the GRBFKAN (IoU↑ Acc↑).

Summary: These results reinforce that GRBFKAN offers clear advantages, especially in challenging scenarios involving complex textures or precise segmentation. We will integrate this extended analysis into the main paper to better justify our architectural choices.

Q2: Computational Efficiency.

Response: We appreciate the reviewer's concern regarding the processing time. We agree that efficiency is important, especially for interactive scenarios, and would like to elaborate on this from the following perspectives:

1. Justified Trade-off for Quality: KaRF’s relatively longer runtime stems from its use of Kolmogorov-Arnold Networks (KANs), which significantly enhance nonlinear expressiveness. This is critical for achieving superior editing fidelity, multi-view consistency, and fine-grained color control. As shown in Figure 5 (main paper) and Figure 4 (supplementary material), KaRF reduces artifacts like color bleeding, which are often observed in faster methods such as LAENeRF and IReNE.

2. Favorable Comparison with SOTA Methods: While slower than LAENeRF, KaRF is faster than other SOTA editing methods such as PaletteNeRF and the recent 3DGS-based method ICE-G. This positions it as a competitive choice when editing quality is prioritized. Meanwhile, this indicates that KaRF's processing time is within a reasonable range, offering a practical balance between speed and quality.

3. Quality-Speed Trade-off: As shown in Tab 3, We conduct additional experiments to demonstrate tunable speed-quality trade-offs. Reducing the local recoloring stage from 3k to 2k iterations shortens runtime to ~6.5 minutes with only marginal PSNR/SSIM drops. This makes KaRF viable for semi-interactive use cases.

Tab 3: Quality-speed trade-off (PSNR↑ SSIM↑).

Q3: The improvement in segmentation performance is relatively limited.

Response: Thank you for this insightful question regarding our segmentation performance. We agree that when looking solely at the average metrics in Table 1 (main paper), the quantitative gains from KaRF may seem modest.

However, we emphasize that in the context of local color editing, the visual quality and boundary sharpness of the segmentation is far more impactful than marginal gains in overall averages. As shown in Figure 4 (main paper), SA3D often yields blurred or imprecise boundaries, while KaRF produces sharper, texture-preserving masks—essential for preventing color bleeding.

Crucially, segmentation in our framework is not an end goal but a critical intermediate step for enabling high-fidelity local editing. It is this high-precision segmentation that allows KaRF to perform nearly perfect, isolated recoloring. The profound downstream impact of this precision is quantified in Table 2 (main paper): the MSE in non-edited regions for our method is an order of magnitude lower than that of all competing methods. This dramatic reduction in error directly stems from our accurate boundaries, which prevent the color bleeding that plagues other approaches.

In summary, while the metric deltas are small, their downstream impact on editing quality is substantial and meaningful.

Q4: Limitations are discussed in the supplementary materials.

Response: Thank you for raising this important point. We completely agree with your assessment. In preparing the initial manuscript, due to a strict page limit, we had to make difficult decisions regarding content placement. To dedicate more space in the main paper to our core methodology and experimental results, we moved the discussion on limitations, certain finer experimental details, and additional visualizations to the supplementary material. And we commit to moving the limitations section from the supplementary material back into the main paper. This change will ensure that all readers can easily access our discussion on the scope of the work and potential avenues for future research.

Thank you for your positive feedback on our paper. We will now provide our responses to the weaknesses and questions you identified.

Q1: Smart combination of existing work, but with great results.

Response: Thank you for recognizing the strength of our approach. While we build on powerful techniques from multiple domains, our contribution lies in their careful integration, which is tailored for local color editing in neural radiance fields. Specifically, our two-stage KAN-based architecture, with its residual adaptive gating design and palette-based loss, brings notable improvements in segmentation precision and local recoloring quality. We appreciate your positive feedback on our results.

Q2: Processing times seem too high for becoming practical.

Response: Thank you for the reviewer’s insightful comment regarding the processing time. We agree that efficiency is crucial for practical applications, and we address this concern from the following perspectives:

1. Trade-off Between Quality and Speed: KaRF is designed to prioritize high-fidelity editing, multi-view consistency, and fine-grained control. This naturally incurs higher computational cost compared to some faster methods (e.g., LAENeRF, IReNe), but yields superior editing quality, as shown in Figure 5 (main paper) and Figure 4 (supplementary material). We believe this trade-off is justified, especially for applications demanding precision.

2. Source of Computational Overhead: The primary cost arises from using Kolmogorov-Arnold Networks (KANs), which offer significantly greater nonlinear expressiveness than MLPs. As validated in our ablation studies, this contributes directly to better segmentation and local recoloring performance, albeit with added computation.

3. Reasonable Compared to SOTA Methods: While our method takes longer than LAENeRF, its processing time is favorable when compared to other SOTA editing methods. KaRF is faster than both PaletteNeRF and the recent 3DGS-based method ICE-G. This indicates that the processing time of KaRF is within a reasonable range, offering a practical balance between speed and quality.

4. Potential for Acceleration: KaRF is flexible and allows for quality-speed trade-offs. By reducing the local recoloring iterations from 3k to 2k or 1k, we can significantly reduce runtime (down to ~6.5 min) while maintaining respectable quality, as shown in Tab 1. This suggests good potential for deployment in semi-interactive scenarios.

Tab 1: Quality-speed trade-off (PSNR↑ SSIM↑).

Q3: The method is sound, presented and evaluated properly. More than a question, it is good to see existing methods for 2D to be repurposed for NeRFs in such a clean and successful way.

Response: We are truly grateful for such a positive and encouraging assessment of our work. We are delighted that you found our method to be sound and presented and evaluated properly. Your observation perfectly captures our core motivation: to repurpose powerful, existing 2D methods for NeRFs in a clean and successful way. We hope our method can build a bridge between mature 2D technologies and emerging 3D representations.