Record demonstrations¶
A VLA learns from demonstrations: camera images, the robot's state and the action taken at every step. urdf2vla records
them in Isaac Sim with collect_dataset and keeps only the successful episodes.
Before recording, run the two preparation steps once:
urdf2vla configure_vla_interface --project projects/myrobot # what the VLA sees and outputs
urdf2vla setup_teleoperation --project projects/myrobot # checks the hand moves as commanded
What gets recorded¶
| Contents | |
|---|---|
| Images | Every camera (default: one wrist per arm, plus left and right), 256×256 RGB, at the control rate (20 Hz by default) |
| State | Every arm's joint positions + one gripper value per gripper (0 = closed, 1 = open) |
| Action | Every arm's joint targets + every gripper command — what the controller was told to do |
| Task text | The instruction for this episode, naming its object, e.g. "put the apple in the bin" |
Each episode puts one of the task's objects on the table at a random position. It counts as a success when the
object rests inside the bin and the gripper has let go,
held for a few steps. Episodes that fail are discarded, so the dataset contains only good demonstrations. Recordings go
to datasets/<name>.hdf5 (demos by default; choose another with --dataset).
Three ways to record¶
A scripted controller picks up the object and puts it in the bin, with a new random object and position each episode. On a dual-arm robot it uses the nearer arm and keeps the other arm open at its initial pose. This is the quickest way to get a dataset and to check the whole pipeline.
It tries up to 3× --num-demos episodes (--max-attempts to change). Failed attempts are summarized in
build/collect_report.yaml with a snapshot in snapshots/collect_fail_*.png, so you can tell whether grasping or
placing went wrong.
Drive the robot yourself while watching Isaac Sim through WebRTC livestreaming — the simulator runs in the container, and you see it in NVIDIA's streaming client.
- Install the Isaac Sim WebRTC Streaming Client (see "Livestream Clients" in the Isaac Sim documentation).
- Connect it to the machine running urdf2vla (
127.0.0.1on the same machine, otherwise its IP address). - Click into the streamed view so it receives your keys, and drive with the keys below.
Recording is paced to real time, so the robot moves at the speed you see.
No GUI at all: the simulation runs headless and a second terminal sends your keys over a local socket.
urdf2vla collect_dataset --project projects/myrobot --device socket --num-demos 10
# in a second terminal, once the first one is waiting:
urdf2vla teleop-keys
Useful over SSH. You don't see the scene, so it's mostly for quick tests; watch the saved snapshots.
Keys¶
| Keys | Moves the hand |
|---|---|
| W / S | forward / back (along the robot's front, +x) |
| A / D | left / right (±y) |
| Q / E | up / down (±z) |
| Z / X, T / G, C / V | roll, pitch, yaw |
| K | open / close the gripper on the active arm |
| N | switch the active arm (dual-arm robots) |
| R | discard this episode and start again |
| Esc | finish recording |
The hand moves in small steps (8 mm and 0.04 rad per control step); change pos_step and rot_step under
overrides.teleop in project.yaml if it feels too slow or too fast. An episode is saved automatically as soon as the
object has been placed in the bin. The instruction for the episode is printed when it starts.
Multiply demonstrations with Isaac Lab Mimic¶
Isaac Lab Mimic
turns a few demonstrations into many: it splits each source demonstration into grasp the object and put it in the
bin, moves each part to where the object and the bin are in a new episode, runs it, and keeps the attempts that
succeed. generate_mimic runs the whole thing for a task with one object and a container:
# record the source demonstrations yourself: run `urdf2vla teleop-keys` in a second terminal
urdf2vla generate_mimic --project projects/myrobot --source-device socket --source-demos 5 --trials 100
# or let the scripted controller record them (a quick first test)
urdf2vla generate_mimic --project projects/myrobot --source-demos 10 --trials 100
- Source demonstrations —
--source-demossuccessful episodes, recorded with the poses Mimic needs (datasets/demos_mimic_src.hdf5). By default the scripted controller records them; with--source-device socket(keys fromurdf2vla teleop-keys) orkeyboardyou record them yourself. Only episodes that put the object in the container and let go of it count. Press R to retry an episode and Esc to stop. - Generation — Mimic tries
--trialsnew episodes, each with the object at a new random position, and keeps the successful ones (datasets/demos_mimic_gen.hdf5, actions only). - Recording — each generated episode is replayed from its start state with the cameras on and saved like any
other recording (
datasets/demos.hdf5), with the same checks ascollect_dataset.
The result reports both numbers: how many of the --trials Mimic generated successfully, and how many of those
replayed into the dataset. --trials is a fixed number of attempts, not a target number of successes.
With the Franka example (task: {objects: [color_cube], container: klt_bin}), 10 source demonstrations gave
75 successful episodes out of 100 trials, all 75 recorded with images, with the cube spread over the whole
spawn area (±27° of yaw) — in about 45 minutes on one RTX 5080 (3 min source, 23 min generation, 18 min recording).
Two-arm robots¶
Each arm is its own end-effector for Mimic. The arm that grasps the object in your source demonstrations grasps and
places in the generated ones. The other arm keeps the pose it had in the source demonstrations and is not moved with
the object (that would push an idle hand into the container). All source demonstrations must use the same arm: if
you record with both arms, generate_mimic stops and asks you to record each arm separately.
This is how to build a dataset for robots whose scripted controller doesn't do the task well, such as OpenArm or Dual
Scorpion. Set the robot and the scene up (launch_sim and setup_teleoperation pass), record a handful of
demonstrations yourself, and let Mimic multiply them. With 5 source demonstrations (recorded by the scripted
controller as a stand-in for a person) and 20 trials, the idle arm's hand stayed still (0 mm) in every generated
episode:
| Robot | Working arm | Generated | Recorded with images |
|---|---|---|---|
Two Frankas (examples/dual_franka) |
left | 15 of 20 | 15 |
OpenArm (examples/openarm) |
left | 20 of 20 | 20 |
Arms are switched with N while you teleoperate (urdf2vla teleop-keys).
How many demonstrations?¶
Start with 50 scripted demonstrations to get the whole pipeline working end to end. For a policy that works from many different objects and positions, plan for more — typically hundreds. Human (keyboard) demonstrations add variety that a scripted controller doesn't have.
Checks during recording¶
collect_dataset checks the recording itself, because these problems are invisible in the numbers and ruin training:
| Check | Fails when |
|---|---|
camera_frames_updated |
a camera image doesn't change between two steps (a frozen camera) |
first_frame_fresh |
an episode's first image still shows the previous episode's scene |
label_replay |
replaying only the recorded actions doesn't reproduce the recorded motion |
See Troubleshooting for what to do when one fails.