Probably the only viable alternative that is still reliable would be single sideband (SSB), but that would be a infrastructure change as well.
If you consider what is required to do a digital encoding including handshaking, synchronization and the like, you are less likely to have a less reliable system. If in the midst of an AM transmission you get a static crash, it likely deletes one word. In a digital system, you might well have to resync, like a modem.
So some of these systems need to be very reliable under all sorts of harsh conditions. The questions I would ask, is 1) do the satellites already exist 2) how many of them and 3) are they visible everywhere there are airflights? and finally 4) how expensive is this network to maintain?
My guess is that the cost of such a network is a lot greater than a few black-box searches. Additionally, for really difficult crashes, they retrieve every possible piece of the airplane, at an expense that makes finding the black box seem small: http://en.wikipedia.org/wiki/TWA_Flight_800