Github has public keys available, so the server owner could compare the key you sent to those - and that would identify your github username.

Whether that's a concern or not is an open question, of course.