It's like the same answer to the question why did ads appear so quickly on the internet in the 90s. It's just that they found new ways to monetize seemingly random / normal everyday things.
Absolutely. I have no idea how that process has evolved over time. For all I know maybe they use Axiom/Flock to do this now but back in my day it was all closed circuit and heavily protected information. Even doing a records request on someone would not disclose this information as their sole purpose was to identify national security risks.
There were lots of institutions decades ago that were not what they presented themselves publicly as being, in one form or another; the difference was that the alternative media outlets that would have allowed interested people to disseminate this information widely were less developed than they are today.