The way to drive out middlemen is to allow competition to make the market more efficient, which will cause people to get outraged at the people who profit from making it more efficient (same thing that happens to HFT firms, which have demonstrably reduced spreads).

The real problem is a lack of supply which is mostly political (although some areas have geographical constraints that make it harder to build... but that's largely solvable by building vertically)