>I've heard the idea that it's temperature change which causes things to cross the barrier, and the fridge is the place in your kitchen with the most stable temperature.

Correct, but it is not temperature change in general, it is when that change is great enough to cause condensation. Everytime you take something out of the fridge it generally will cause water to condense on the surface because of that temperature change from cold fridge to warm kitchen, same reason you glass of cold water sweats on a hot summer day.

The conventional wisdom here probably changes with region (as conventional wisdom often does) since everywhere does not have the same climate. UK is a rather damp place so fridge is probably safer for much of the year. On reflection I did not think that part through completely, did not take into account everyone having a different local climate.