Anecdotally, this has never been a problem for me. My side project is even registered as a company in the UK and being directors of it has never been an issue for the full time "real" jobs me and the other two people involved have.