Work in progress

Generative Failure-to-Success Bridging for Robot Learning

Junha Hyung*, Hyojin Jang*, Jaegul Choo

KAIST

* Equal contribution

Paper arXiv Code Progress
top row — original simulator rollout bottom row — failure + generated recovery + success
A robot misses its grasp, closing the gripper on empty air. Instead of discarding the episode, we ask a generative model to produce the bridge — the missing segment of experience that carries the arm from the failed state back onto a successful trajectory. Outside the highlighted window the two rows are identical; inside it, the bottom row is entirely model output (71 frames, marked by the red dot).

Abstract

Large robot datasets contain abundant successful demonstrations and informative failures, yet they rarely show how to recover after a mistake. A failure marks a state in which a policy needs help; a successful demonstration reveals a state from which the task can continue. What is missing is the segment that connects the two — and collecting it is expensive, because each recovery has to be re-rolled from the exact state where things went wrong.

We propose Generative Bridging (GB), a framework that converts disconnected failure and success trajectories into synthetic recovery demonstrations. A vision-language model parses task stages and detects failure-induced regressions; a retrieval module proposes semantically valid and locally compatible success suffixes; a pretrained inverse dynamics model generates the missing recovery actions; and an action-conditioned world model reconstructs the intermediate observations required for frame–action policy training.

Our central hypothesis is that carefully filtered synthetic bridges can improve downstream recovery and task success without requiring exact reconstruction of each real-world scene in a simulator. The novelty is not endpoint interpolation alone, but a complete, uncertainty-aware framework for turning offline failures and successes into training data that teaches a robot how to get back on track.

This page reports early progress on the observation-completion component: a world model fine-tuned on RoboCasa regenerates a 71-frame recovery window from a single conditioning frame and an action sequence. The full pipeline and downstream policy evaluation are in progress.

System design

Method overview

The pipeline separates semantic decisions — when recovery is needed and where it should return — from generative decisions — how to move, and what observations the policy should see.

01Parse stagesVLM labels task state, entities, relations, confidence.
02Detect failureFind semantic regression or a critical failure event.
03Retrieve suffixesSelect advanced stages from successful demonstrations.
04Rank pairsCheck target identity, local geometry, view, horizon.
05Generate bridgeIDM proposes bounded action sequences, up to 96 steps.
06Complete framesWorld model produces chunk-start or dense observations.
07Train policyConfidence-weighted real and synthetic supervision.

Semantic filtering removes impossible pairings before expensive generation; endpoint consistency filters the generated candidates afterward. Steps 5–6 are what the results below exercise.

Problem formulation

Given offline failure and success trajectories for the same task family, we synthesize a valid recovery segment and use it as additional policy supervision. Each trajectory contains visual observations o, proprioceptive state q, and robot actions a.

p = (oftp, qftp),   s = (osts, qsts),   Â = GIDM(p, s),   |Â| = L ≤ 96 A post-failure prefix p is bridged to a compatible success suffix s by a pretrained inverse dynamics model, over a bounded horizon.
Ô1:L = W(op, Â),   τ̃rec = {(ôℓ, q̂ℓ, âℓ)}ℓ=0L−1 A world model W completes the missing observations, so the bridge can supervise an observation-conditioned policy.
Where recovery starts and ends

Choosing the two endpoints

Endpoint selection is a semantic problem before it is a generative one. Getting it wrong makes every downstream generation worthless.

Prefix: the first stable state after failure

The prefix is not the moment of error itself. It is the first stable state after a consequential failure, when a corrective policy could reasonably take control. For every short video window we predict a structured semantic state — task stage, target object, achieved relations, failure event, calibrated confidence — smooth those labels over time, and detect a drop from the highest reliably achieved stage.

Reach · S1→ Grasp · S2→ Lift · S3→ Drop→ Reach · S1  /  Prefix
Filmstrip of frames 177 to 195 showing the failed grasp and the stitch seam, in the agentview and wrist cameras
A concrete prefix in our RoboCasa episode. Frames 177–195, agentview (top) and wrist (bottom). The gripper closes on empty air beside the potato; the prefix is taken once the arm settles after the failed grasp, not at the instant of the miss.

Suffix: restore the last reliably achieved stage

Suffix selection is a constrained retrieval problem: first require semantic validity, then rank candidates by local compatibility and generative feasibility.

Current · S1⇢ S2⇢ S3 complete  /  S4 entry

Key design rule. If the robot regresses from stage 3 to stage 1, the target should not merely be stage 2. It should restore the last reliably achieved state — stage 3 completed — or enter stage 4, reducing the chance of replaying the same failed transition.

  1. Semantic gateSame task family, same manipulated entity, and target stage at or beyond the pre-failure stage.
  2. Local compatibilityCompare object pose, gripper pose, robot configuration, and target-relative geometry where available.
  3. View-invariant similarityUse object-centric visual embeddings; tolerate background differences that do not alter contact geometry.
  4. Horizon feasibilityReject candidates predicted to require more than 96 steps, or to violate action and workspace limits.
  5. Generated endpoint testRun top-ranked pairs through the IDM and world model; retain candidates whose predicted endpoint best matches the suffix.
s* = arg mins ∈ 𝒞(p) [ λgoaldϕ(ôL, os) + λctxdctx(p,s) + λdynJdyn(Â) + λlenL ] Subject to semantic-stage admissibility, matching target identity, action bounds, and L ≤ 96.
Action and observation synthesis

Generating the bridge

A pretrained IDM supplies the actions; an action-conditioned world model supplies the observations an observation-conditioned policy needs.

Condition

Two endpoints

The failure prefix anchors the recovery start; the success suffix specifies a state from which task execution can resume.

Generate

Variable horizon

Produce a bounded sequence of up to roughly 96 action steps, allowing recoveries longer than a typical 32-action policy chunk.

Filter

Quality before volume

Prefer endpoint agreement, smooth and bounded controls, collision-risk proxies, and model confidence over unfiltered synthetic scale.

World-model completion

The IDM gives actions but not the intermediate visual observations. For a policy predicting H actions from one frame — e.g. H = 32 — the world model needs to supply only one conditioning frame per chunk, so generation can run at chunk boundaries rather than densely.

ô(k+1)H = W(ôkH, âkH:(k+1)H−1),   k = 0, …, ⌈L/H⌉−1 Chunk-boundary generation supplies exactly one conditioning frame for each policy action chunk. Dense per-step frames are optional.

Interpretation boundary. Endpoint interpolation is not by itself proof of physical recoverability, and if the same world model both generates observations and ranks bridges, endpoint consistency may just reflect model bias. "Recovery" is claimed at the policy level; generated bridges are candidate supervision, filtered by consistency signals and calibrated against held-out real transitions.

Early result · step 06

Regenerating the recovery window

One conditioning frame, then actions only — the model never sees another real frame.

To test the observation-completion component in isolation, we fine-tune Cosmos3-Nano into an action-conditioned forward-dynamics model on RoboCasa kitchen manipulation, jointly generating three camera views on a single canvas so the produced frames stay mutually consistent.

The episode below splices a failed rollout to a successful one at frame 185 — a hand-constructed instance of the prefix/suffix pair the retrieval stage is meant to find automatically. We hand the model frame 184 alone, plus the 71 recorded actions of the recovery window, and let it generate the intervening frames as five chained 16-action chunks. Each chunk conditions on the last frame the previous chunk generated, so error accumulates with no ground-truth resets.

Frames 165–264, zoomed in around the transition. Top: simulator ground truth. Bottom: model output. Arm pose, potato position and the timing of the re-approach track the reference; the visible gap is mostly sharpness. Past the recovery window (f256 onward) the rollout keeps running without ground truth and drift becomes apparent.
SettingPSNR (dB)Frames above baseline
Repeat conditioning frame15.78—
Generated recovery (ours)20.7971 / 71

Measured over the 71 recovery frames against the simulator rollout, on the three-view strip. Per-chunk PSNR: 21.34 · 20.73 · 21.73 · 20.64 · 17.85 — degradation is confined to the final chunk, consistent with autoregressive drift.

What this number is and is not. The failed rollout for this task is present in the world model's fine-tuning set, so the scene and its initial state are not unseen. The result should be read as action-following fidelity within a known scene, not as generalization to novel scenes, and not yet as evidence that the bridge is physically recoverable. A held-out evaluation is part of the ongoing work below.

Implementation

Setup

Three views generated jointly, so the bridge is usable as multi-view training data.

The world model is trained in forward_dynamics mode: every action token is clean conditioning and the loss is vision-only rectified flow. The three RoboCasa cameras — two third-person agentviews and the wrist camera — are tiled into one 640×640 canvas so a single generation pass keeps them geometrically consistent with each other, then sliced back apart downstream. Actions are 7-D arm deltas re-encoded to a 10-D [Δpos, rot6d, gripper] layout and quantile-normalized.

The source episode as recorded: three RGB views over their aligned depth maps. The red dot marks the recovery window. The failed grasp is clearest in the wrist camera around f178–191, where the fingers close to 0.2 cm on empty air beside the potato.
End-effector trajectory and gripper aperture analysis
End-effector displacement, path geometry and gripper aperture across the episode. The shaded band is the recovery window: the gripper reopens, the arm descends toward the object, and the re-grasp closes at f224 — after which the aperture stops short of closing, the signature of an object between the fingers.
Learning objective

Policy training

Train on real demonstrations and accepted synthetic bridges jointly, while explicitly accounting for synthetic-data uncertainty.

ℒ = 𝔼τ∼𝒟real[ℓπ(τ)] + β · 𝔼τ̃∼𝒟GB[w(τ̃) ℓπ(τ̃)] β controls synthetic-data exposure; w combines VLM confidence, pair compatibility, endpoint agreement, and dynamics quality.

Curriculum

Begin with short, high-confidence bridges; introduce longer and more diverse recoveries after the policy learns stable local correction.

Mixture control

Tune real-to-synthetic sampling and cap repeated suffix patterns to avoid synthetic distribution dominance.

Confidence weighting

Down-weight uncertain VLM parses, large context mismatch, weak endpoint agreement, and high-action-cost bridges.

Evidence strategy

Evaluation plan

The primary claim is downstream policy improvement. Bridge-level metrics establish plausibility and diagnose failure modes, but do not replace real task evaluation.

StudyQuestionPrimary metrics
Downstream policy Does GB improve behavior after mistakes? Train identical policies with and without generated recovery data; evaluate on held-out scenes and induced perturbations. Task success, post-failure recovery, interventions, time-to-recovery
Bridge consistency Do generated actions approach the selected suffix? World-model endpoint comparison, proprioceptive agreement, action-limit and smoothness checks. Endpoint distance, control cost, rejection rate
Controlled execution Are bridges executable when full state is available? Run the pipeline on native simulation tasks where scenes and contacts are known — not as exact replay of real scenes. Execution success, collision rate, horizon error
Human audit Do samples look semantically and physically plausible? Blind comparison of random, retrieved, and GB-filtered trajectory clips. Preference, plausibility, target correctness
Baselines
  1. Real data onlySuccess and failure trajectories without augmentation.
  2. Replay / oversamplingReweight real failure neighborhoods without synthesis.
  3. Random semantic suffixSame stage rules, no local compatibility or endpoint filter.
  4. Nearest-state retrievalGeometry or visual similarity without generative validation.
Ablations
  1. No VLM stage parsingUse scalar progress or temporal heuristics.
  2. No local matchingKeep only task and stage constraints.
  3. No endpoint filterAccept every IDM sample within the horizon.
  4. No world-model framesAction-only or single-frame supervision where supported.
  5. Bridge horizonCompare ≤32, ≤64, and ≤96 steps.
Testable hypotheses

Research questions

  1. RQ1

    Does semantic stage reasoning select better recovery start and target states than scalar progress or visual nearest neighbors?

  2. RQ2

    Does Generative Bridging improve downstream task success specifically after perturbations and naturally occurring failures?

  3. RQ3

    Which filtering signals — local geometry, predicted horizon, endpoint agreement, or control cost — best predict useful synthetic data?

  4. RQ4

    Are world-model-generated observations necessary, and what temporal density is sufficient for action-chunk policies?

  5. RQ5

    How does the quality–diversity tradeoff change as the synthetic-to-real training ratio and bridge horizon increase?

Critical view

Risks & limitations

The proposal is strongest when it makes narrow, falsifiable claims: generated bridges are useful candidate supervision, not guaranteed real-world recovery trajectories.

Failure / success environment mismatch
Use object-centric semantic gates and target-relative geometry. Treat background changes as nuisance only when they do not alter relevant contacts or obstacles.
IDM physical implausibility
Enforce action and workspace limits, smoothness constraints, independent endpoint scoring, and a high rejection threshold. Report acceptance rate.
World-model hallucination
Generate only the frames required by the policy interface, calibrate on held-out real transitions, and ablate action-only supervision.
VLM stage ambiguity
Use task-specific stage ontologies, temporal windows, confidence thresholds, and a small human-labeled calibration set.
Unreachable recovery within 96 steps
Predict horizon feasibility, reject overly distant targets, or decompose a bridge into verified semantic subgoals.
Model-circular validation
Separate proposal generation from scoring where possible; include human audits, held-out dynamics checks, and real policy outcomes.

Status

This is an early-stage project page. The result above is a single-episode proof of concept for one component of the pipeline; the rest is planned work.

BibTeX

@misc{hyung2026bridging,
  title  = {Generative Failure-to-Success Bridging for Robot Learning},
  author = {Hyung, Junha and Jang, Hyojin and Choo, Jaegul},
  year   = {2026},
  note   = {Work in progress}
}