I am uncomfortable about it too (and haven't installed any in my home), but I'm not sure what to do about the exact same issues from smart phones that we carry with us all the time: they can easily listen in on our conversation, in addition to knowing when that conversation happened and where.

The home pods are a bit of a drop in the bucket when you think about "smartphone coverage around the world".

What options are there?