You say that as if you're not helping Google with anything else. Every search, every plus, every rating, every interaction helps Google with its business. And in majority of cases, users reap the benefits too and house numbers is one of those—better geographical knowledge is always good. Whether it can be abused is irrelevant to this specific matter. Every piece of information about anything can be abused.