Maybe just take away that protection? Where I live nobody is losing a house for having the wrong color mailbox, but there are certain historical protected areas where beauty is an externality — you can’t demolish a historic house in the middle of the beautiful downtown