You will only find out that something went wrong 30 years later and that is horrifying for people that still have asbestos in their homes and offices.
It also requires a pretty systematic exposure to the stuff, which is why mostly (factory) workers and the neighborhoods surrounding those factories were seeing the most severe cases. Plus the process takes years, which is why it stayed unnoticed for so long (minus the cover ups).
Health hazards aside it's a pretty remarkable product that was cheap at the same time. Exactly why it was used for pretty much everything.
Don't get me wrong, the stuff is bad, but I do think that some, if not most people react almost histerically when they know they are in the vicinity of asbestos. Living in a house with asbestos shingles for instance poses very little health hazards, on the condition that the shingles aren't being disturbed by anything that causes friction. Of course, you don't control your neighbor's roof...
If you have small quantities at home that you need to dispose of make sure it's wet and don't break or saw it.
We live in nano age, what are the chances that Graphene or any other nano material can have similar effect over long period of time?
Not protecting people in the entire product lifecycle was criminal negligence on an industrial scale. They knew.