At the Risk of Forgetting