This is something I don’t believe we’ve really figured out in terms of QAing stochastic processes. Doing QA on normal code is binary...did it work or not? Doing a QA on a model is a lot more difficult. You need to think about Type I and Type II error. If I were a product manager thinking about wake words, I’d want my Type I error to be as small as possible...to the point where I’d accept a pretty high Type II error.

Lately I’ve been seeing Alexa wake on accident more often. It’s like someone isn’t thinking critically about what kind of Type I error they’re willing to accept.