Personalized Vision via Visual In-Context Learning

Thank you for your detailed review and acknowledgement of the efficiency and significance of research problem. We address your feedback below; please let us know if there are further comments or concerns.

Our unique contribution

We would like to clarify how PICO differs from prior Visual ICL methods:

1. Prior work does not address test-time generalization.

   Existing Visual ICL methods (Painter [1], LVM [2]) focus on solving predefined tasks seen during large-scale pretraining. They do not evaluate, nor demonstrate, generalization to new tasks or new objects at test time, which is the core goal of ICL.

2. We target the personalized vision setting for the first time.

   We explicitly study the dual challenge of adapting at test-time to user-defined objects and tasks under severe data constraints. This is a scenario unaddressed by previous methods.

3. Our solution is tailored to this setting.

   We leverage a powerful generative prior (FLUX) and finetune it on VisRel, our compact yet highly diverse dataset to enable visual ICL. The method spans both recognition and generation tasks, covering varied personalized subjects and tasks definition.

Details about VisRel dataset construction

1. Mask colors in segmentation.

For each four‑panel sample we randomly sample RGBA for each category instance, keeping colors consistent within a sample but varying across samples. This prevents color‑overfitting and forces the model to learn visual relations rather than specific palettes. See Supp. Fig. 1 for an example, where different visual prompts produce semantic, dichotomous, and matting outputs, etc. for the same image.

2. Why the current 27 tasks?

We aim to exhaustively cover all major computer‑vision I/O formats (pixels, masks, keypoints, depth, etc.). To achieve balanced coverage, we organized tasks along two intuitive axes: (i) Semantic complexity: low-medium-high and (ii) Spatial locality (local-object-global). Each cell of this 3 × 3 taxonomy is populated with representative tasks, ensuring a richly connected visual relation space that supports strong ICL generalization at test time.

3. Task-type ablation.

Thank you for this insightful suggestions. Our VisRel dataset is designed to balance diversity across task types (low‑level, physical/geometric, semantic, generative) to holistically support personalization. We performed additional ablation studies to assess the impact of each task type on personalized image segmentation (Main Sec. 4.2). For each category, we removed three tasks (reducing training data by 30 samples):

The results show:

• Physical/geometric tasks (depth/reshading/keypoints3D) consistently help: Removal reduces performance. 3D/spatial reasoning improves object boundary awareness.

• Generative tasks (doodling/relighting/line art colorization) are critical: Removal causes big performance drop. These high-semantic/local tasks teach object-aligned editing essential for segmenting user-specific objects.

• Low-level tasks (deblur/dehaze/low light enhancement) are neutral but valuable: Removal shows small changes. While not directly beneficial, they don’t harm performance, validating our inclusive design.

• Semantic perception tasks (stuff segmentation/object detection/affordance) can conflict: Removal improves results. We hypothesize that class‑level labels may suppress fine instance‑level distinctions, which are essential for personalised segmentation.

We will integrate these results and discussion into the final version.

Writing clarifications

Thank you for the suggestions, we will revise the paper to improve clarity and readability:

• Dataset (Sec 3.1): We will clearly state the dataset size and list all 27 tasks directly in the method section, rather than referring readers to the supplementary material.

• Model pipeline (Sec 3.2): The four images are concatenated into a single 2x2 grid in RGB space before the VAE encoder. The Cell (2, 2) is the black placeholder "X". in the latent space, we apply noise to its latent E(X), and only this quadrant is supervised. Lines 140‑155 will be rewritten to eliminate ambiguity.

Main experiment results aginas SegGPT

While PICO’s absolute metrics in Table. 1 lag behind SegGPT [3] and PerSAM [4] on some tasks, this comparison overlooks a key advantage: PICO achieves comparable SOTA results using only 40 labeled segmenetation samples, representing only 0.016% of SegGPT’s 254K training data.

In personalized vision scenarios, data is limited, and the large-scale labeled data is often infeasible to obain. PICO's efficiency is highlighted here:

• Matches SegGPT/PerSAM despite orders-of-magnitude less supervision,
• Outperforms all prior personalized segmentation methods,
• Surpasses existing visual ICL baselines.

This demonstrates PICO’s strong potential for low‑data regimes where annotation scarcity is the norm.

References:

[1] Wang, Xinlong, et al. "Images speak in images: A generalist painter for in-context visual learning." CVPR, 2023.

[2] Bai, Yutong, et al. "Sequential modeling enables scalable learning for large vision models." CVPR, 2024.

[3] Wang, Xinlong, et al. "Seggpt: Segmenting everything in context." ICCV, 2023.

[4] Zhang, Renrui, et al. "Personalize Segment Anything Model with One Shot." ICLR, 2024.

Thank you for the follow-up questions. We clarify each point below.

Definition of "test-time generalization"

We apologize for the eariler overstatement. Painter [1] and LVM [2] do show certain forms of generalization (e.g., open-vocabulary classes, logical puzzles). However, their evaluations remain within predefined tasks seen during training, e.g,. Painter on depth estimation, semantic segmenatins, deraining, etc.; LVM on foreground segmentation, object detection, colorization.

In contrast, our work specifically targets the test-time personalization settings, where both the object (e.g., personalized segmentation) and the task definition (e.g., novel compositional tasks) are new at inference time. We provide a systematic, quantitative evaluation of this capability, demonstrating that our method outperforms existing Visual ICL approaches under this stricter and more practical setting.

Clarification of the "placeholder" (Cell 2, 2)

``` 
# Inference phase   
Grid: [A  A′
       B  X ]            # X = constant black (or any)
1. Replace X with pure Gaussian noise ε in latent space
2. Condition on the three clean quadrants and iteratively denoise X
3. Decode → predicted B′

# Train phase
Grid: [A  A′
       B  B′ ]            # B′ = *ground-truth* target

1. Encode full grid → latent z₀
2. Add flow-matching noise: zₜ = (1-t)·z₀ + t·ε
3. Restore the three context quadrants from z₀
    – TL (A), TR (A′), BL (B) are clean
    – BR (B′) stays noisy      ← only region the model must denoise
4. Loss computed **only** on BR quadrant (noise − z₀)
```

Please let us know if you have further questions, or if there is anything else we can address for you to consider raising our score. Thank you again for your time and effort in reviewing our paper!

Thank you for your valuable review and recognizing the value of the personalized-vision setting, thorough experiments and clear presentation. We address your feedback below; please let us know if there are further comments or concerns.

Construction of the VisRel dataset

The VisRel dataset is carefully designed to span both inter-task and intra-task diveristy, inspired by how text corpora expose LLMs to many latent "tasks". Our key insight is that a robuts visual ICL model should learn a unified visual-relation space, enabling interpolation and composition of transformations at test-time. Additional dataset construction details are provided in Supp. A.

No, it doesn't contain the same instance or task type during training. To avoid train-test leakage:

• object categories: Test instances in personalized segmentation datasets (PerSeg/DOGS/PODS/PerMIS) are rare and never appear during training.
• task types: during training, we use single tasks (e.g., inpainting, style transfer) standalone, while at test-time we access compositional tasks (e.g., simultaneous watermark removal and stylization).

Adapting to new tasks

The model inherently has in-context adaptation capabilities, effectively interpreting new tasks from visual examplars provided at inference. If the new task is completely outside VisRel's coverage (i.e., outside of the visual-relation space seen during training), we observe the model's adaptability is limited (as noted in the limitation Sec). Nevertheless, the retraining is efficeint and inexpensive, requiring only minimal finetuning (approximately 5-10 examples for one task) on a single GPU.

Spatial position of the placeholder

We fixed the placeholder position during training and inference. To evaluate the impact of placeholder positioning, we conducted the experiment by changing the curernt horizontal layout (Top–Bottom, TB) to a vertical arrangement (Left–Right, LR), using identical datasets. The results below show it has minimal impact on overall model performance, and in some cases, improves personalized segmentation metrics. The grid layout (or, essentially, the positional embedding) has little impact, but the learned visual-relation space drives performance.

(1) Personalized image segmentation (Main Sec. 4.2)

(2) Personalized test-time task generalization (Supp C.1). Novel compositional tasks: A: Deraining with inpainting; B: Inpainting with stylization.

Thank you for your valuable review and recognizing the value of the personalized-vision setting, the breadth of experiments and the clarity of our paper. We address your feedback below; please let us know if there are further comments or concerns.

Difference to Visual Prompting

PICO fundamentally differs from VP [1] from its motivation, method design to task scope.

• Motivation & Method design.

  • VP, as a pioneer visual ICL work, provides a unifed format for classic predefined vision tasks as image-to-image via inpainting. PICO, while using the same grid-like format, instead targets open-ended, user-defined tasks at test-time, including personalized tasks and objects.
  • To achieve this, PICO leverages the strong generative prior with compact LoRA finetuning on proposed intra and inter task-diverse VisRel dataset (315 four panel samples). The training, small but diverse, teaches the model to learn the visual relations implied by exeamplar pair, enable strong in-context generalization that VP does not demonstrate.

• Task scope. Unlike VP's focus on simpler, standardized tasks (e.g, edge detection, foreground segmentation), PICO addresses complex tasks and personalized scenarios, including dense prediction, creative image doodling, personalized segmentation, and novel compositional tasks, etc.

Thus, PICO is not am incremental enhancement over VP but rather extends the paradigm from fixed, narrow tasks to flexible, personalized vision with robust in‑context generalization capabilities.

Additional baselines

We appreciate your suggestions and have evaluated OmniGen [2] and LVM [3] under our personalized vision settings:

(1) Personalized image segmentation (Main Sec. 4.2)

(2) Personalized test-time task generalization (Supp C.1). Novel compositional tasks: A: Deraining with inpainting; B: Inpainting with stylization.

PICO consistently outperforms both baselines by a big margin. Both rely on large-scale general pretraining and struggle to interpret examplar instructions effectively at test-time:

• OmniGen often produces blue segmentation masks, ignoring the visual instruction.
• LVM confuses the depth map and segmentation sometimes, and fails to deal with the compositional tasks.

In contrast, our VisRel's diversity trains the model to decode the transformation implicit in any A → A' visual exemplar, yielding superior performance across all personalized tasks. We will include these additional experimental results in the final version.

Minor grid-size question

Our grid resolution is 1024x1024, with each sub-image at 512x512, aligned with FLUX.1’s native resolution to balance quality and computational efficiency. The base model FLUX.1 can support higher resolutions (e.g., 1920×1080 or 1536×1536) with great generation quality.

References:

[1] Bar, Amir, et al. "Visual prompting via image inpainting." NeurIPS, 2022.

[2] Xiao, Shitao, et al. "OmniGen: Unified image generation." CVPR, 2025.

[3] Bai, Yutong, et al. "Sequential modeling enables scalable learning for large vision models." CVPR, 2024.

Thank you for your thoughtful review and acknowledgement of PICO's simplicity, strong results, and potential utility. We address your feedback below; please let us know if there are further comments or concerns.

Novelty against Visual Prompting and Analogist

1. Difference from VP [1].

   VP provides a unifed format for vision tasks and is evaluated only on predefined, simple tasks, with limited in-context reasoning ability. PICO targets open‑ended, personalized tasks at test time (including user-defined objects and tasks) and demonstrates robust in‑context generalization ability beyond the training set.

2. Difference from Analogist [2].

   Analogist is training-free and therefore restricted to Stable Diffusion’s RGB output space. It cannot handle dense or discriminative prediction tasks, whose outputs lie in other spaces such as depth maps, normal maps or segmentation masks. while PICO's lightweight finetuning to repurpose the generative prior, unlocking dense/discriminative tasks (e.g., depth/normal estimation, pose estimation), while also supporting the RGB-space generation tasks.

Claim of "a new paradigm for personalized vision"

Prior “personalized vision” work focuses on personalized objects within a single task (e.g., instance segmentation) [3, 4, 5]. In this data-scarce scenario, previous method (PRPG [3]) typically use generative priors to synthesize more object-specific data for training personalized representations, which is very computational expensive. PICO instead treats the generative model as a visual in-context learner, allowing it to adapt at test time to both personalized objects and tasks without additional training.

Meanwhle, existing visual ICL methods are either (i) aimed primarily for semantic-driven image manipulation (e.g., Analogist [2], InstaManip [6]), which cannot tackle perception tasks, or (ii) predefined multi-task models (e.g., VP [1], Painter [7]), which are large-scale trained on classic CV tasks, and lack the in-context generalization needed in personalized setting. So neither camp supports the object- and task-level personalization.

Therefore, we argue that PICO introduces a new paradigm for personalized vision, supporting both user‑defined tasks and personalized objects, from perception to generation tasks, within a single, flexible framework.

Subjective language revisions

Thank you for the feedback. We will tone down the claims and ensure the language is evidence‑driven. Subjective phrases will be replaced with objective, quantifiable statements tied directly to our experiments. For example: “unprecedented flexibility” → “broad flexibility”; “promising results” → “state‑of‑the‑art or competitive results”.

New insights taking away

1. Task diveristy > sheer data volumne for visual ICL.

   More data of the same task mainly improves memorisation; broader task coverage drives true generalization during test-time. This is crucial for building adaptable, sample-efficient vision systems. For example, compare task balance and data efficiency:

   Method	# Tasks	Segmentation Examples / Total
   Painter [7]	7	138K / 192K
   Ours (PICO)	27	40 / 315

   Painter’s training data is large but imbalanced dataset. In contrast, PICO uses a compact but diverse dataset, yielding strong generalization to unseen personalized tasks where Painter fails.

2. Generative pretraining provides transferable unified features for perception and generation.

   While prior works (e.g., Marigold [8], GeoWizard [9]) unleash generative priors for dense prediction, they focus only on perception and do not support generation. To our knowledge, few studies explore unifying both discriminative (e.g., segmentation, depth) and generation (e.g., stylization, doodling) within a single generative framework, enabled via lightweight finetuning.

Apples‑to‑oranges comparisons

We agree this is a valid concern. Since our work targets a new setting, i.e, evaluating the ability to solve unseen personalized tasks (both object- and task-level), it is quite challenging to find perfectly aligned baselines.

We tried our best to ensure fairness by selecting the strongest available baselines and covering a broad range of methods. For example, in Personalized image segmenation: we include large-scale pretrained models (PerSAM [10], SegGPT [11]), the prior personalized segmenation approaches (PRPG [3], PDM [5]), and visual in-context methods (VP [1], Painter [7]).

Finally, our ablation studies to isolate contributions within our method, and to attribute performance gain to our method.

References:

[1] Bar, Amir, et al. "Visual prompting via image inpainting." NeurIPS, 2022.

[2] Gu, Zheng, et al. "Analogist: Out-of-the-box visual in-context learning with image diffusion model." ACM TOG, 2024.

[3] Sundaram, Shobhita, et al. "Personalized Representation from Personalized Generation." ICLR 2024.

[4] Nguyen, Thao, et al. "Yo'Chameleon: Personalized Vision and Language Generation." CVPR, 2025.

[5] Samuel, Dvir, et al. "Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval." NeurIPS, 2024.

[6] Lai, Bolin, et al. "Unleashing in-context learning of autoregressive models for few-shot image manipulation." CVPR, 2025.

[7] Wang, Xinlong, et al. "Images speak in images: A generalist painter for in-context visual learning." CVPR, 2023.

[8] Ke, Bingxin, et al. "Repurposing diffusion-based image generators for monocular depth estimation." CVPR, 2024.

[9] Fu, Xiao, et al. "Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image." ECCV, 2024.

[10] Zhang, Renrui, et al. "Personalize Segment Anything Model with One Shot." ICLR, 2024.

[11] Wang, Xinlong, et al. "Seggpt: Segmenting everything in context." ICCV, 2023.

Thanks for providing a detailed response!

The differences from VP and Analogists are clearer to me now. I can see how you might consider those methods not to count as "personalization" since they are more limited in the kinds of test time transformations they can adapt to. However, I still would tone down the language around "new paradigm" and general claims of methodological novelty. It's totally fine if it's an extension of existing ideas. I think the critical novelty is that these ideas work really well for "personalized vision", in ways that weren't demonstrated in the prior papers. Again, I'd personally be more favorable to this paper if the framing were more like "Standard methods do really well on personalized vision, calling into question if we need entirely new methods."

I'll consider raising my score. I see the value in the work even though I disagree a bit with the framing.