All it takes is being someone who
* Writes three books on particle physics by the age of 14
* Gets into, then drops out of Eton because of boredom. He instead ...
* Gets into, then drops out of Oxford because of boredom. He instead ...
* Gets into graduate school at Caltech
* Receive a PhD
* Do all of the above by the age of 20
See? Easy!
Despite my sarcasm above, the above shouldn't be (entirely) taken as discouragement. A one-in-a-century genius like Wolfram can do the above before turning 21. It's not impossible that you are the, say, one-in-a-decade genius who can, without being so young, as impendia said impress a professor with your insight. That professor might be able to, in turn, see whether his department's graduate program can admit someone on a non-degree basis (not unusual at all), taking a class or two at a time for a grade but not in a formal program.
If you're not "good enough", you'll likely be able to apply the classes toward a bachelor's degree. If you impress the right people in the right way and if the university does not formally restrict admission to a graduate program to bachelor's degree holders, it's not impossible that you could be formally admitted. Or perhaps a professor will be able to recommend you to a program at a university that does not have such a restriction. There are a lot of "if"s, but it's also not completely impossible.
PS - Yale Law School admitted someone without a bachelor's degree a while back. He was a very successful corporate executive who late in life decided to go to law school and did very well on the LSAT.