... but you are completely within your rights to remove them. I don't see an ethical issue with that (I mean I do see the ethical issue, but after thinking about it I do not believe removing them is unethical).

By removing you might by a little (or no) time before they find out.

Is it worth it?

Is what you have to say going to be a huge tip off?

At the end of the day, this is your startup, your job, your company and livelihood and completely your call to make.

my2c