However, way less than I'm afraid of any random idiot human who may or may not have passed a driver's test on his 5th attempt in some random state, driving at whatever diminished capacity (tired, distracted, drunk, etc.) in the worst weather conditions, who is angry at the car in front of me for some reason, driving on the highway near me.
You can easily have the worst of both worlds, where somebody disables the autonomy to engage in some road rage, then re-enables it, but without driver awareness features (lots of videos of people doing this) so they can have a nap and not take over when the autonomy disengages.
So you can easily have all the disadvantages of autonomous cars and idiot reckless driving.
Only time we'll be free is if there's ever L4/L5 autonomous driving, which seems a long way off.
edit: even "full self-driving" in Tesla's are L3, since you need to be available to take over, driver awareness is enforced, etc.