There is a step where the auditor will observe you entering a correct and an incorrect password to enter the systems. If your org is depending on SSH key passphrases for this step and you get the wrong person in front of the auditor (the one without the passphrase on their key) then you just failed the audit. The more steps you fail, the deeper down the rabbit holes they go with each step. If they see you are not failing, it will be a check-box exercise. Each auditor is a little different of course; but generally speaking, this is true.