I live in a decent apartment in a big south/west German city, I have a 60k+ salary and yes, I pay a lot in taxes and for rent. There's NO way I have the same quality of life as an unemployed family, while they might get some money from the state it's nowhere close to a 60k salary.
Now, where I do see the problem is that most of the housing is owned by people renting it out (check out Germany in this list [1]). My whole building is owned by the same guy who I guess lives quite a relaxed life out of my rent, while producing nothing. If you want to fight the increase in living costs, I'd start there.
I have no problem with feeding and sheltering refugees.
[1] - https://en.wikipedia.org/wiki/List_of_countries_by_home_owne...