feat: held-out validation during training#77
Open
ZouzouWP wants to merge 1 commit into
Open
Conversation
Until now, training loss was the only signal available during a run — no way to tell a policy was starting to overfit without stopping and evaluating it separately afterward, by which point the ideal checkpoint was already steps behind.
## What changed
- New "Validation" section in training's advanced config: a single toggle drives `dataset.eval_split` (fraction of episodes held out from training) and `eval_steps` (evaluation cadence) together, since `lerobot` requires both or neither.
- Backend parses `step N: eval_loss=X` lines (emitted by `lerobot_train` whenever `eval_steps > 0`) into a new `eval_loss` field, exposed on both the live training status and the persisted metrics history (so the curve survives a page reload).
- The monitoring chart plots `eval_loss` as a second, sparser line alongside the training loss, so overfitting becomes visible while training is still running instead of only in hindsight.
```mermaid
flowchart TB
A["Held-out episodes\n(e.g. 10%)"] --> B["Periodic evaluation"]
C["Training loop"] --> D["Training loss\n(every step)"]
B --> E["Validation loss\n(every N steps)"]
D --> F["Monitoring chart"]
E --> F
```
## Testing
Implemented and code-reviewed against the log-parsing regex and the chart's dedupe/merge logic, but **not yet run end-to-end** with `eval_steps > 0` against a real training job — needs a validation pass to confirm the parsed values match `lerobot_train`'s actual output format before merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Until now, training loss was the only signal available during a run — no way to tell a policy was starting to overfit without stopping and evaluating it separately afterward, by which point the ideal checkpoint was already steps behind.
What changed
dataset.eval_split(fraction of episodes held out from training) andeval_steps(evaluation cadence) together, sincelerobotrequires both or neither.step N: eval_loss=Xlines (emitted bylerobot_trainwhenevereval_steps > 0) into a neweval_lossfield, exposed on both the live training status and the persisted metrics history (so the curve survives a page reload).eval_lossas a second, sparser line alongside the training loss, so overfitting becomes visible while training is still running instead of only in hindsight.flowchart TB A["Held-out episodes\n(e.g. 10%)"] --> B["Periodic evaluation"] C["Training loop"] --> D["Training loss\n(every step)"] B --> E["Validation loss\n(every N steps)"] D --> F["Monitoring chart"] E --> FTesting
Implemented and code-reviewed against the log-parsing regex and the chart's dedupe/merge logic, but not yet run end-to-end with
eval_steps > 0against a real training job — needs a validation pass to confirm the parsed values matchlerobot_train's actual output format before merge.