Cool post. I thought this was the most interesting point:

"As an outside agency, we can’t (and shouldn’t) make this decision for Zidisha. However, it IS our job to inform them well enough to make it. The ROC curve is abstract and hard to interpret for this purpose, so we translated its information into a plot that directly measures the trade-off at every possible threshold value."

So many of the standard techniques in machine learning assume an objective function that's different from what the end-user actually cares about. Good practitioners like Everett know to translate between the algorithm's loss function (e.g., squared error) and the end goal. I'm surprised there's not more research to let ML algorithms optimize the ultimate loss function directly!