feat(datasets): browse a local dataset's episodes in-app#68
Conversation
A dataset on disk currently can't be viewed at all. The viewer is a deep-link to the hosted lerobot/visualize_dataset Space, which renders from a Hub repo path, so Landing sends local datasets to /upload instead — you have to push a dataset to the Hub before you can look at what you just recorded. /edit-dataset has been a routed placeholder. Fill it in: pick a dataset, page its episodes, watch every camera frame by frame, with a film strip and an aggregate joint-motion trace. Backend — episode_media.py reads the v3.0 layout (meta/episodes/*.parquet + videos/) directly and decodes with PyAV. It deliberately avoids LeRobotDataset so browsing never builds a robot config or imports torch. No new dependencies: PyAV, numpy, pyarrow and Pillow all arrive with lerobot[core_scripts], which is already required. Notes on two things that are easy to get wrong: - Episodes share an mp4. Locating one resolves the file *and* its from_/to_timestamp window inside it. /dataset-video is therefore addressed by chunk/file, not by episode, so the URL stays stable as you page between episodes: the player seeks instead of reloading, and playback survives the switch. An episode-addressed URL hands the browser a new identifier for the same bytes and silently drops playback on every click. - duration_s is length/fps, not to_timestamp - from_timestamp. An episode accepted early stops its parquet at accept while the camera ran to the configured limit, so the window over-reports the demonstration. The window is still exposed for the transport. Frontend — the Landing picker now opens local/both datasets in the browser rather than diverting to /upload; Hub-only datasets still go to the Space, since there's nothing here to decode. Upload stays reachable from the browse page. Tests build a real (tiny) v3.0 dataset on disk — a genuine PyAV-encoded mp4 plus parquet metadata — rather than mocking the readers: the point of the module is that it agrees with the on-disk format, which a mock can't check.
|
Nice work! Thumbnail index/png misalignment In Suggested fix: keep the index bound to its frame instead of relying on positional alignment. e.g. have Copyright header
Your two follow-up PR offers — both agreed
Everything else, path-traversal guard, chunk/file video addressing, the duration_s = length/fps rule, sync def routes keeping PyAV decode off the event loop, looks very good. |
2 participants
The gap
A dataset that exists on disk can't currently be viewed in LeLab.
The viewer is a
window.opendeep-link to the hostedlerobot/visualize_datasetSpace, which renders from a Hub repo path.Landing.tsxencodes the consequence:handlePickExistingroutessource: "local" | "both"datasets to/uploadinstead. So the dataset you just recorded has to be pushed to the Hub before you can look at it — which is backwards, since checking it is usually why you'd decide whether to keep it./edit-datasethas been a routed 15-line placeholder ("This page is under construction") with no backend. This fills it in.What this adds
Pick a dataset, page through its episodes, and watch any camera frame by frame — with a film strip and an aggregate joint-motion trace, entirely against local files.
GET /dataset-episodesGET /dataset-episodeGET /dataset-frameGET /dataset-thumbnailsGET /dataset-motionGET /dataset-videoinline+ Range-capableThe Landing picker now opens local datasets in the browser instead of diverting to
/upload. Hub-only datasets still hand off to the Space — there's nothing local to decode. This changes existing behaviour, so it's the piece most worth your opinion; Upload stays reachable from the browse page.No new dependencies
PyAV, numpy, pyarrow and Pillow all arrive via
lerobot[core_scripts,feetech,training], whichpyproject.tomlalready requires (core_scripts→dataset→av-dep). Nothing was added.episode_media.pyreads the v3.0 layout (meta/episodes/*.parquet+videos/) directly and decodes in-process with PyAV. It deliberately avoidsLeRobotDataset, so browsing a dataset never constructs a robot config or imports torch — listing and previewing stay cheap.Two details that are easy to get wrong
Episodes share an mp4. Locating one means resolving the file and its
from_/to_timestampwindow inside it./dataset-videois therefore addressed by chunk/file rather than by episode, so the URL stays byte-identical as you page between episodes: the player seeks instead of reloading, and playback survives the switch. An episode-addressed URL hands the browser a new identifier for the same bytes, forcing a reload and silently dropping playback on every click. A test pins this.duration_sislength / fps, notto_timestamp - from_timestamp. An episode accepted early stops its parquet at accept while the camera ran to the configured limit, so the window over-reports the demonstration (a ~2.6s episode reads as the configured 10s). The window is still exposed for the transport.Testing
223 passed(187 existing + 36 new).ruff check,ruff format,tscandeslintare clean, andnpm run buildpasses.The new tests build a real (tiny) v3.0 dataset on disk — a genuine PyAV-encoded mp4 plus parquet metadata — rather than mocking the readers. The whole point of the module is that it agrees with the on-disk format, which a mock can't check. Covered: path traversal, camera/episode resolution, the packed and pre-packing layouts, the
duration_srule above, decode-lands-in-the-right-window,inline+ Range on the video route, and the shared-URL contract.Verified by hand against two datasets:
lerobot/svla_so101_pickplace(50 episodes, 30fps, camerasside/up) and an unrelated in-house one (20fps, three cameras, 7-DoF), so the format assumptions aren't tuned to one producer.frontend/distis intentionally not included —build_frontend.ymlrebuilds it on merge to main.Documentation
README.mdgains a Browse row;CLAUDE.mdgainsdatasets.pyandepisode_media.pyin the feature-module list, including the chunk/file addressing rationale. A stale comment inserver.py("Replay is rendered by the embeddedvisualize_datasetSpace") is corrected — it's a link-out, not an embed, and local datasets no longer go there.Unrelated and not touched: the README's ⏪ Replay — "Re-run any recorded episode" row looks stale.
ba0ddfband8a72410merged record/replay and dropped the/replay-datasetroute, so nothing re-runs an episode on hardware any more. Happy to fix in a separate PR if you agree.Open question
lelab --devhardcodes:8000/:8080with no override, which collides with anything already on:8000. I worked around it locally with a--portuvicorn and the?api=override. Worth a flag? Separate PR if so.Contributed by VibeCuisine. Draft — happy to adjust the Landing behaviour change, the route naming, or the scope before this goes green.