I turn down about 1/2 of consulting offers (two today, which is a record), spending the time on research, general learning, and the non-tech aspects of living.
For me the trick is: I got an agreement from my wife that it is better to remain on top of new tech than to maximize earning. Also, living in a low cost area helps increase flexibility (Carol and I sold our large house near the beach in San Diego, and bought a small economical house in the mountains in Central Arizona).
The great mythologist Joseph Campbell advised his students to "follow their bliss," and I can agree with that as long as you are supporting yourself and family. In life, try to do what makes you happy as often as possible.
Also, +1 for you for not wanting to go into debt.