+100. I did exactly this. After working for 12 years, took a break, learnt 3D game programming and actually shipped a playable 3D sport simulation game on iOS. Enjoyed graphics programming a ton. Also prototyped the game in Common Lisp before porting it to iOS.
End result: I now work as a scientific software developer specializing in 3D visualization all thanks to the graphics knowledge gained in the process of shipping the game.
So I would say once you have covered enough fundamentals, you must ship something you think is hard but just harder and not completely beyond you.
For me the fundamentals are: 1. Multiple programming paradigms 2. Algorithms 3. Data Structures 4. Digital Electronics 5. OS 6. Networking and 7. Math for CS. This will take a lot of time. I invested a lot in 1 - 4, skimmed 5,6,7; shipped a game and things unfolded.