With physical security, it would be more like "I trust Bob isn't going to hit me with a wrench because he told me he wouldn't".
Bob isn't very trustworthy.
Sure, we could encrypt everything ever created by any device at all times. But some sort of communication is sent from one IP address to another IP address. Even if it's encrypted, some form of something was sent.
This is where privacy then becomes a thing. You could start sending random noise out constantly. then when your encrypted data is sent out it looks like the rest of the random noise.
That would be like "I sent out 4999 random messages to 4999 random people and 1 encrypted message to Alice. Now Bob can't work out I sent my encrypted message to Alice".
That's the difference. Privacy is about hiding the existence of something from Bob. Security is about keeping something safe from Bob.