back
4 comments
I think even setting aside the edge device aspect of this, continual learning is quite challenging. Even doing good experiments about continual learning in a single architecture (which kind of require simulating a model's "lifetime" of exposure to a mix of previously-seen and new kinds of data) can be pretty tricky. Making architectural changes _as you learn_ as some of these researchers are discussing seems more challenging still. Aiming to do all of that with low power/compute seems like it's asking quite a lot. Co-designing algorithms, model architectures and hardware simultaneously seems like a moonshot.
I'd argue not developing them simultaneously is like trying to create a robot using anatomical designs from mammalian, avian, and insect domains without modifying the initial organ systems.

Either way it is a challenging problem, but we can't simply re-use existing system integration motifs to develop on-device continuous learning, in my opinion.

It will take a lot of prototypes and “reinventing the biological wheel” to achieve some kind of integration between the ML systems which were previously designed with a very specific task in mind. Truly exciting times.
I'm trying to follow the field closely but I need to make sure my understanding is not outstripped by developments. I'm still trying to build a habit of reading research papers instead of widely spaced bursts when inspiration hits.