Companies where payments are central (e.g., PayPal, Square) end up building some combination of machine learning, investigation tools, a dedicated operations team to review/verify suspicious transactions, and custom logic to look at all sorts of signals correlated with fraud. Often they'll have dozens or hundreds of people working on this.
For everybody else, I'd echo Eran's advice to just outsource this. There are plenty of vendors out there. Here's one list: https://www.merchantriskcouncil.org/index.cfm?pageId=702
If anybody out there is dealing with fraud or chargebacks, my company (Sift Science) provides an API to do exactly the checks Eran's article suggests and a lot more. Even if our technology doesn't apply, I'm happy to just give advice and point people in the right direction. My e-mail is brandon@siftscience.com.