A lot of government debt is held by private entities. You're thinking about the national debt, and here is a long detailed explanation:

http://internationalecon.com/Finance/Fch5/F5-9.php