A data synthesis framework that transforms abundant in-the-wild 2D pose datasets into diverse 3D pose–image pairs, boosting the generalization of 3D pose estimators without costly 3D annotations.
PoseSyn addresses challenging pose cases without direct costly 3D annotations. Starting from an inaccurate pseudo-labeled pose, PoseSyn generates diverse motion sequences around it to produce new image–pose pairs, bridging the gap between the inaccurate pseudo-label and the desired challenging pose while expanding the training distribution with hard examples.
Despite considerable efforts to enhance the generalization of 3D pose estimators without costly 3D annotations, existing data augmentation methods struggle in real-world scenarios with diverse human appearances and complex poses. We propose PoseSyn, a novel data synthesis framework that transforms abundant in-the-wild 2D pose datasets into diverse 3D pose–image pairs. PoseSyn comprises two key components: the Error Extraction Module (EEM), which identifies challenging poses from the 2D pose datasets, and the Motion Synthesis Module (MSM), which synthesizes motion sequences around the challenging poses. Then, by generating realistic 3D training data via a human animation model — aligned with challenging poses and appearances — PoseSyn boosts the accuracy of various 3D pose estimators by up to 14% across real-world benchmarks including various backgrounds and occlusions, challenging poses, and multi-view scenarios. Extensive experiments further confirm that PoseSyn is a scalable and effective approach for improving generalization without relying on expensive 3D annotations, regardless of the pose estimator's model size or design.
Two complementary modules turn 2D pose seeds into realistic, challenging 3D training data.
PoseSyn consists of two main modules. The Error Extraction Module (EEM) identifies challenging poses in the 2D dataset by comparing the GT 2D pose with the 2D projection of the pseudo-labeled 3D pose predicted by a target pose estimator (TPE). The Motion Synthesis Module (MSM) then generates diverse motion sequences around these challenging poses, guided by both a VLM-generated caption and the mis-predicted pose. Finally, a human animation model renders motion-aligned images with varied appearances and backgrounds, which are filtered and used to fine-tune the TPE.
EEM isolates the intricate, dynamic poses where the pose estimator struggles.
(a) Challenging data. EEM identifies intricate and dynamic poses from the in-the-wild 2D pose dataset — exactly the corner cases where the target pose estimator underperforms.
(b) Non-challenging data. In contrast, non-challenging data consists primarily of stationary, static poses. Their reference images are reused for motion-guided rendering.
MSM augments a single mis-predicted pose into diverse motion sequences that approximate the hidden ground-truth challenging pose.
SMG architecture. The proposed Semantic-guided Motion Generation (SMG) augments a mis-predicted pose into motion sequences. An initial motion representation $\mathcal{MR}_{\text{init}}$ is encoded and mapped into a codebook to produce initial motion indices $\mathcal{S}_{\mathcal{MR}}$. A transformer takes both the text embeddings $\mathbf{e}_{\text{text}}$ (from a VLM caption of the challenging image) and $\mathcal{S}_{\mathcal{MR}}$ as input, and autoregressively generates the motion indices for the synthesized motion sequence $\mathcal{M}_{\text{C}}$. Combining text and initial-pose guidance resolves the ambiguity of text-only generation and better targets the challenging pose.
Consistent generalization gains across three target pose estimators and six real-world benchmarks.
3DCrowdNet trained with only real data exhibits inaccurate 3D pose predictions with limited generalization. Baselines (PoseGen and Ours-N) show limited gains, while our approach achieves more accurate pose predictions across diverse real-world datasets with various backgrounds and occlusions, challenging poses, and multi-view scenarios. Red boxes highlight incorrect predictions in models trained with real-only data and baseline methods.
Select a benchmark to view per-dataset results. 3D benchmarks (3DPW, EMDB, CMU, HuMMan) report MPJPE / PA-MPJPE (mm, lower is better); 2D benchmarks (LSPET, JHMDB) report PCKh0.6 (higher is better). Across all three TPEs (3DCrowdNet, HybrIK, 4DHumans), PoseSyn consistently outperforms both baselines (PoseGen and Ours-N), yielding 6–14% MPJPE and 5–9% PA-MPJPE improvements over the real-only model.
| Method | Mean↓ ± Std↑ | Min↓ |
|---|---|---|
| (a) $\hat{J}^{\text{3D}}$ (mis-prediction) | 181.7 ± 0.0 mm | 181.7 mm |
| (b) w/o $\mathcal{MR}_{\text{init}}$ | 222.3 ± 36.4 mm | 151.1 mm (−16.8%) |
| (c) Ours (full MSM) | 209.3 ± 36.5 mm | 140.8 mm (−22.5%) |
By synthesizing plausible pose variants, MSM produces at least one pose closer to the true challenging pose than the single naive mis-prediction. Omitting the initial motion representation $\mathcal{MR}_{\text{init}}$ raises both mean and min error, confirming its vital role in targeting challenging poses.
| Model (EEM) | Train | Metric | 3DPW | EMDB | CMU 171204 | CMU 171206 | HuMMan | Mean |
|---|---|---|---|---|---|---|---|---|
| w/o EEM | FT | MPJPE↓ | 80.2 | 114.2 | 107.1 | 109.0 | 94.9 | 101.1 |
| PA-MPJPE↓ | 50.5 | 71.1 | 72.0 | 71.1 | 65.1 | 66.0 | ||
| HybrIK | FT | MPJPE↓ | 79.3 | 113.5 | 103.5 | 106.2 | 94.9 | 99.5 |
| PA-MPJPE↓ | 50.0 | 70.4 | 69.4 | 68.8 | 64.6 | 64.6 | ||
| 4DHumans | FT | MPJPE↓ | 78.6 | 112.7 | 103.0 | 106.6 | 93.2 | 98.8 |
| PA-MPJPE↓ | 50.0 | 69.9 | 68.4 | 68.4 | 64.2 | 64.2 | ||
| 3DCrowdNet | FS | MPJPE↓ | 78.8 | 112.4 | 103.0 | 106.2 | 93.5 | 98.8 |
| PA-MPJPE↓ | 49.5 | 69.9 | 68.2 | 67.9 | 63.7 | 63.8 | ||
| 3DCrowdNet | FT | MPJPE↓ | 77.4 | 111.0 | 101.0 | 105.0 | 93.1 | 97.5 |
| PA-MPJPE↓ | 48.9 | 68.3 | 67.3 | 67.9 | 62.3 | 62.9 |
Effect of the pose estimator used inside EEM on final TPE (3DCrowdNet) performance. Compared to (1) excluding EEM entirely, (2) using a different estimator (HybrIK, 4DHumans) in EEM, and (3) using the same 3DCrowdNet in EEM but training from scratch (FS), using the same pre-trained TPE with fine-tuning (FT) yields the best results — confirming that focusing synthesis on each TPE's own problematic poses maximizes generalization. FT: fine-tuning; FS: from scratch.
A single reference image augmented with diverse challenging poses, and a single problematic pose rendered across varied appearances, viewpoints, and backgrounds.
@inproceedings{yang2025posesyn,
title = {PoseSyn: Synthesizing Diverse 3D Pose Data from In-the-Wild 2D Data},
author = {Yang, ChangHee and Song, Hyeonseop and Choi, Seokhun and Lee, Seungwoo and Kim, Jaechul and Do, Hoseok},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2025}
}