I sucessfully have been doing this for more than seven years: the trick is decent but fair pricing per time unit; also you have to budget and calculate a buffer realistically. Yes, you also need downtime and if you are stressing about that you are most likely billing too low or are simply too anxious/greedy (all things that can be fixed).

Enjoy the freedom, at last that's what you have optimized for.