> not that unlikely that the same image could match both.

That seems like a statement that needs to be justified. I suppose one could apply a 50% dropout, construct adversarial pairs on that network, and then run them though the the other half-network to get an idea of the frequency of match on both.

My intuition is that the probability of match of an adversarial pair on a second, independent network should be close to the probability of a non-adversarial collision. Robust adversarial perturbations are hard enough when only considering class labels, nevermind hidden state