Salvatore Calcagno, Matteo Pennisi, Federica Proietto Salanitri, Amelia Sorrenti, Simone Palazzo, Concetto Spampinato and Giovanni Bellitto
This is the official PyTorch implementation for paper "Dream2Learn: Structured Generative Dreaming for Continual Learning".
|
Scoreboard
|
 |
 |
 |
 |
 |
 |
|
Ladybug
|
 |
 |
 |
 |
 |
 |
|
School bus
|
 |
 |
 |
 |
 |
 |
Continual learning requires balancing plasticity and stability while mitigating catastrophic forgetting.
Inspired by the concept of human dreaming as a source of internal simulation and knowledge restructuring, we introduce Dream2Learn (D2L), a framework in which a continual classifier leverages its internal representations to guide the generation of structured synthetic experiences and uses them for self-improvement.
Rather than reconstructing past data as in generative replay, D2L enables a classifier to create novel, semantically distinct dreamed classes that are coherent with its learned knowledge yet do not correspond to previously observed data. These dreamed samples are produced by conditioning a frozen diffusion model through soft prompt optimization driven by the classifier itself. The generated data are not used to replace memory, but to expand and reorganize the representation space, effectively allowing the network to leverage internally synthesized auxiliary concepts during training. By integrating dreamed classes into continual training, D2L proactively structures latent features to support forward knowledge transfer and adaptation to future tasks. This prospective self-training mechanism conceptually relates to the role of sleep in consolidating and reorganizing memory, turning internal simulations into a tool for improved generalization. Experiments on Mini-ImageNet, FG-ImageNet, and ImageNet-R demonstrate that D2L consistently outperforms strong rehearsal-based baselines and improves forward transfer, achieving positive values in several of the evaluated settings, confirming its ability to enhance adaptability through internally generated training signals.
Code release coming soon.
This project is under construction.