Continual learning and what it would mean for AI safety

Understanding continual learning

Continual learning is a vague concept. Some of the things you will hear around it are:

  • A model which learns at runtime (deployment) vs designtime (training) qualifies as continual learning.
  • Continual learning is learning from new data while while retaining performance on old data (which is generally how it is defined in literature), as opposed to something like non-stationary online learning where we don’t care about old performance.
  • It is arbitrary and just depends on the frequency that we update model weights (so having an automatically LLM manage it’s SKILL.md files would not count).
  • It is the set of efforts aimed at breaking past the feasible horizon of current techniques (so SFT, RL post-training, and context management all qualify).

For the purpose of this post, I am going to consider a continual learning as a model whose weights will be updated regularly after deployment (e.g., more than once per month). This does seem to be on the roadmap for frontier labs and in fact, Cursor already has some kind of continual learning in deployment (model weights are updated as much as every 5 hours).

We should also note that on short AGI timelines, continual learning likely looks very different from recursive self-improvement (RSI). Recursive self-improvement looks more like solving research taste in frontier models and closing the loop on AI R&D so that model N builds model N+1 which builds model N+1 and so on. It is very uncertain whether this could begin in the next ~5 years, but it could plausibly lead to a vastly superhuman model. On the other hand, continual learning alone (model N is built to never stop learning) almost certainly won’t lead to a vastly superhuman model.

Implications for AI safety

On one hand, continual learning may fix out-of-distribution mis-generalisation, thereby improving alignment insofar as human preferences are difficult to specify for every situation so having a model that actively seeks and ‘continually learns’ human preferences would be good.

On the other hand, it presents a number of complications for AI safety:

  • There is grow·ing evi·dence in model psychology research that LLMs are persona simulators and fine-tuning on bad data can activate toxic personas (emergent misalignment). Continual learning could accidentally activate a bad persona, and this could plausibly self-reinforce through mechanisms like exploration hacking. I suspect this is unlikely to happen catastrophically in practice but I do think some form of alignment drift is more likely.
  • Continual learning would probably be bad for interpretability. If model weights are updated regularly then we would naively expect interpretability to become less tractable.
  • It would also be very bad for adversarial testing and auditing. Currently, frontier labs conduct rigorous safety testing prior to deployment of their models. If the models are updated frequently after deployment, this becomes a lot harder.

What could we work on to prepare for this:

  • Create a measure of distance from an evaluated state of the model, which can trigger evaluation if reached.
  • Work on methods to predict how a model update will affect alignment.
  • Work on safety evaluations and monitors designed for a continual learning system.