This is mostly about the heat absorption properties of walls. Another property of walls, which they do allude to, is the protection from wind, which can be even more important in more hostile environments. When wind hits a wall (or obstacle) there is a period of space behind the wall where the shelter is actually higher than the wall. I think this is called the Law of the wall[0] in fluid dynamics. If you place your fruit trees the optimal distance from the wall you can provide maximum shelter, and if you place the 2nd tree an optimal distance from the 1st tree then the 1st can shelter the 2nd, and so on. My parents were able to use this theory to grow some of the only fruit trees on a windswept island notable for its almost complete lack of any sort of trees.
So while a walled garden has something of a negative connotation in technology, in horticulture it seems much more positive. Interestingly, looking at the definition of walled garden on wikipedia "A walled garden is a garden enclosed by high walls for horticultural rather than security purposes"[1] (emphasis mine).