>If anything I'd expect the lack of activity to cause your immune system to be hyper-aware and you'd get less sick

That's not really how it works afaik. The immune system can only respond quickly and effectively to the things it knows. If you spend 3 years outside society and you come back, your system will be 3 years behind on disease evolution. When you finally do get exposed, you'll have a much stronger immune response because it's an unknown threat.

Compare this to a immune system that has also never seen the latest variant, but has seen the one before -- it'll be much more effective at recognizing and fighting the threat early, and the person might not even notice they were exposed.