It depends a lot on yourself. Math is really like a sport, I was doing it each and every day as part of a Physics/Mathematics major. Whereas in the same class that I was in, many people passed calculus, linear algebra etc, but failed programming courses.

Programming might be easier to get into though