You are looking at this from the wrong pov. The objective of solitary living is to cut yourself off from "society" and its demands/interactions. The ensuing mental calm and peace (there is "joy" in this) is what "hermits" are striving for.