The way i think of it is via three concepts viz. "Randomness", "Cause and Effect", "Correlation (and not causation)". Randomness assuming true uncaused nondeterministic, by definition cannot be modeled by deterministic mathematics. Cause-and-Effect in the real world is actually aggregate multiple causes giving rise to multiple effects and so trying to follow a single cause to effect presupposes you can have knowledge of every single parameter (i.e. nothing hidden) in all the causes and the same for all the effects. Correlation problem arises naturally from a combination of the above two.
Ultimately then, in the Universe; does true randomness exist or is everything deterministic and we just cannot know all parameters involved?
The philosophical text "Yoga Vasistha" uses the following example to illustrate the above; a crow alights on a palm/coconut tree and a fruit falls; is it cause-and-effect or a random occurrence? Can we ever know?
This is why all mathematical models are a simplification and we always make simplifying assumptions on what to ignore when constructing a model and agree to accept the model when the error delta of the result is small enough for our needs. Hence assumptions like "fair dice", "fair coin" etc. in probability theory.
Note that this is not to say we cannot get close to Reality; all of Science is proof that we can, but that there will always be that gap/error delta between Reality and our Model of it. For many cases it may not matter eg. all our technology today, but in some very important cases it does eg. Consciousness.
You might find the following books useful in this regard;
1) Ivar Ekeland has a series of popular mathematics books viz. a) Mathematics and the Unexpected b) The Broken Dice, and other Mathematical Tales of Chance c) The Best of all Possible Worlds: Mathematics and Destiny which i have found pretty interesting and thought-provoking in this regard - https://en.wikipedia.org/wiki/Ivar_Ekeland Read the first one in particular.
2) Timothy Gowers' Mathematics: A Very Short Introduction is more focused on Concepts/Abstraction/Models and hence relevant here.