This is wrong in certain cases. The candidate may have significant contribution to an open source project or may have shipped an open source project as an owner.
Why should that candidate not refuse? You already have the code in front of you. If you as an employer cannot take the time out to go through that and evaluate what you would otherwise with your take home assignment, then something is not right.
This one size fits all rigid process won't work. Given a process, the employers must learn to be flexible, all the while expecting employees to do so, is insane. So certain qualified candidates saying 'someone else will hire me' is not arrogance, but fact.