Many people refinanced in the last few years to take advantage of low rates and take money out of their homes.