A hybrid data-synthesis framework that couples T2I scene diversity with context-aware I2I editing (VRAIN) to boost long-tailed instance segmentation on LVIS.
We present TMI, a hybrid data synthesis framework that couples text-to-image (T2I) generation for scene diversity with context-aware image-to-image (I2I) editing for realistic, accurate rare-class supervision — achieving state-of-the-art long-tailed instance segmentation on LVIS.
Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we propose a hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity.
Overview of our hybrid data generation and training framework. We combine two complementary paradigms: (i) T2I generation for diverse scenes over all categories $C_\text{all}$, and (ii) I2I editing (VRAIN) for rare categories $C_\text{I2I}$ with accurate instance-level supervision. (iii) A student $M_\text{student}$ jointly learns from both $\mathcal{D}_{\text{T2I}}$ and $\mathcal{D}_{\text{I2I}}$. For T2I images $I_m$, an EMA teacher $M_\text{teacher}$ generates refined pseudo-labels $\tilde{A}_m^\text{teacher}$, merged with offline labels $A_{m}^\mathcal{P}$ to form the final supervision $A_m$. As training proceeds, the teacher adapts to the T2I domain, progressively improving initially noisy pseudo-labels.
A two-stage framework that inserts rare-class instances naturally, then rigorously verifies every edit before it becomes supervision.
Our two-stage framework ensures high-fidelity I2I editing. (i) Place: A VLM proposes a semantically coherent instruction ($inst^*$) for inserting a rare class ($c^*$) that fits naturally within $I_\text{real}$. An instruction-based editor $\Phi_\text{I2I}$ then synthesizes $I_\text{edit}$. (ii) Verify: $I_\text{edit}$ is then validated. The new instance is localized via SSIM difference and open-vocabulary detection, semantically confirmed by a VLM filter, and masked using SAM. Finally, an annotation updater resolves occlusions with $A_\text{real}$, integrating the new instance to produce the trustworthy final annotation $A_\text{edit}$.
Qualitative examples of the VRAIN verification stage. (a) A helmet is added instead of the requested headset ($c^*$), creating a semantic mismatch, and is thus filtered out; (b) the target headset is correctly added, passing the verification.
| Method | APbox | APmask | APrbox | APrmask |
|---|---|---|---|---|
| Real-only | 47.5 | 42.3 | 41.4 | 36.8 |
| DI2I only (w/o verification) | 47.4 | 42.5 | 39.5 | 36.0 |
| DI2I only (VRAIN) | 48.1 | 43.3 | 42.9 | 39.5 |
We compare I2I data with and without VLM-based filtering. Removing the verification step degrades both overall and rare-class AP — even falling below the Real-only baseline on rare classes (APrbox 41.4 → 39.5) — whereas VRAIN's verified data improves it (→ 42.9). This confirms the VLM filter is crucial for retaining semantically correct, relevant instances.
Consistent state-of-the-art gains across backbones, with especially strong improvements on rare categories.
| Backbone | Method | T2I | I2I | APbox | APmask | APrbox | APrmask |
|---|---|---|---|---|---|---|---|
| ResNet-50 | Real-only | – | – | 34.5 | 30.8 | 24.0 | 21.6 |
| MosaicFusion | ✓ | 34.1 | 30.4 | 24.4 | 22.5 | ||
| DiverGen | ✓ | 35.1 | 31.2 | 25.6 | 23.8 | ||
| X-Paste | ✓ | 36.7 | 33.0 | 29.6 | 27.8 | ||
| Ours | ✓ | ✓ | 38.1 | 34.0 | 33.9 | 31.7 | |
| Swin-L | Real-only | – | – | 47.5 | 42.3 | 41.4 | 36.8 |
| MosaicFusion | ✓ | 47.7 | 42.8 | 41.3 | 37.5 | ||
| DiverGen | ✓ | 49.6 | 44.2 | 44.5 | 39.8 | ||
| X-Paste | ✓ | 50.1 | 44.4 | 48.2 | 43.3 | ||
| Ours | ✓ | ✓ | 50.7 | 45.2 | 49.1 | 44.0 |
Our hybrid T2I-I2I approach consistently outperforms all baselines: the Real-only model, the T2I-only method (MosaicFusion), and copy-paste I2I methods (DiverGen, X-Paste). This holds across both ResNet-50 and Swin-L backbones, delivering particularly strong gains on rare-class metrics.
| Method | APbox | APmask | APrbox | APrmask |
|---|---|---|---|---|
| Real-only | 47.5 | 42.3 | 41.4 | 36.8 |
| MosaicFusion | 47.7 | 42.8 | 41.3 | 37.5 |
| DiverGen (rare-target) | 47.3 | 42.2 | 34.5 | 31.2 |
| DI2I only (VRAIN) | 48.1 | 43.3 | 42.9 | 39.5 |
Among methods generating data exclusively for rare categories, an ironic situation arises: DiverGen, despite explicitly targeting rare classes, catastrophically underperforms even the Real-only baseline on rare categories (APrbox 41.4 → 34.5). This contradictory result highlights that naive copy-pasting harms performance due to its poor contextual coherence and realism, encouraging the model to overfit to pasted instances rather than learning true representations. In contrast, our rare-class-focused I2I approach (VRAIN) successfully boosts rare-class performance (41.4 → 42.9), benefiting from high visual fidelity and precise instance-level labeling.
Examples of pseudo-label adaptation in T2I images. As the teacher model updates, it adapts to the T2I domain and enhances pseudo-label quality — improving masks and labeling previously missed objects (e.g., quesadilla, telephoto lens) for rare categories.
Final pseudo-labels for T2I images. Teacher-generated pseudo-labels (red) are merged ($\oplus$) with offline pseudo-labels (blue) to form the final supervision. As the teacher adapts to the T2I domain, it successfully labels rare classes such as dove and walrus.
| Method | APbox | APmask | APrbox | APrmask |
|---|---|---|---|---|
| Real-only | 47.5 | 42.3 | 41.4 | 36.8 |
| $M_\text{teacher}$ only | 49.4 | 43.9 | 47.9 | 43.0 |
| $M_\mathcal{P}$ only | 49.9 | 44.9 | 45.3 | 41.7 |
| $M_\text{teacher}$ and $M_\mathcal{P}$ | 50.3 | 45.1 | 47.5 | 43.8 |
We evaluate the impact of different pseudo-label sources. The offline labeler $M_\mathcal{P}$ provides stable supervision for common categories, the EMA teacher $M_\text{teacher}$ adaptively improves rare-class coverage, and combining both leverages their complementary strengths.
Additional rare-class insertions produced by our I2I editing pipeline.
Each example shows, in order, the original image $I_\text{real}$, the original annotation $A_\text{real}$, the edited image $I_\text{edit}$, and the final annotation $A_\text{edit}$ (after VLM verification and SAM mask integration). VRAIN inserts rare-class objects naturally while preserving annotation quality.
@article{song2026tmi,
title = {TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation},
author = {Song, Hyeonseop and Choi, Seokhun and Do, Hoseok},
journal = {arXiv preprint arXiv:2607.08201},
year = {2026}
}