Heat. It gets really hot the deeper you go. There are a lot of mines already where it's very difficult for humans to exist for very long.

Also, availability. There's a lot of stuff in the earth. But after 4 billion years gravity happened. All elements have different densities. Turns out a lot of the metals that are really, really useful also turn out to be pretty dense. Think uranium and platinum as examples. Most of these elements have sank into the deep mantle and the core over time. The bits we find left over aren't even a taste of how much is collectively in the earth but can't get to. On a side note, it's been theorized that the massive amounts of uranium that sank to the core has been powering the earth for billions of years. Basically, the earth is a giant fission reactor.

The third point is that there's a lot more stuff up there than down here. If you had to pick a direction to mine, in the long term it's going to be easier and more profitable to mine the stuff in space than to dig deeper into the ground.