In our experience, you can stop most fraud without putting up roadblocks for your users. Every site is different, but to give an example, we were able detect 90% of fraud for a site with a huge fraud problem without requiring any extra verification from the users.
The really key penalty to avoid is what is called an "excessive chargeback program," which usually triggers for chargeback rates that exceed 1%. You initially get a warning, and if you can't get your chargeback rate down, your payment processor has the right to shut you off. If you're in an excessive chargeback program, then I'd definitely recommend "playing it safe."
But otherwise, I think slimming down your payment form and carefully measuring the effect on fraud is almost always a smart business move.