Since the point of a substantial number of his articles is that quick and dirty usability testing is the only reliable way to make usability decisions I don't think you can reasonably fault him here.
I don't think your account correctly characterizes his approach to usability. I think it correctly characterizes the perception of his approach to usability in many who dismiss him.
That is, he doesn't say "These results are for our site, you should test it yourself for yours. Our findings may not be relevant for your situation." after he's talked about findings or made recommendations.
And he doesn't address situations that are directly in opposition of the findings he does report -- e.g., he has never covered sales letters as I described.
When he does write about "do it yourself" user testing, he does it in separate articles that are only about that topic. They are not situated in the broader context.
He is obviously a very smart person so I don't think that this is accidental.
And he only writes those DIY pieces once or twice a year. Generally speaking his company sells 3-day usability testing workshops for $23,000.
So, let's be realistic here.
Here's an example of one of the rare times when he discloses the actual test data: http://www.useit.com/alertbox/9710a.html
Usually when he makes claims about, say, readability, no examples are offered. I was shocked to find this one.
You'll notice that nowhere in his explanation of the findings does he conjecture that maybe reading comprehension is low because it's bloody boring AND poorly written, and nobody gives two shits about Nebraska's dubious tourist attractions.
He says maybe it's because the language is "promotional." But, again, deathly boring and irrelevance to user's lives doesn't come into his analysis picture at all.
And he doesn't tell you to test your own content, either.
I have all his books and I've read the vast majority of his articles, too. You can dismiss me as somebody who merely has a "perception" problem, but that doesn't change the effort I've put into it.
PS - He reviewed a book on manipulating customers, through fiddling with neuroscientific knobs, thusly:
---
I highly recommend that you read this [book] for two reasons:
* The book's indispensable design advice will grow your business.
* You must teach your children to recognize this new class of manipulation.
---
So, it's cool when you're only manipulating OTHER people's kids.
I don't know why people think of him as if he's some kind of folk hero.
Don't you think that the selling of $23,000 usability testing workshops might constitute an incentive for him to push for more hands on usability testing rather than the blind acceptance of his findings?
But you said that he is a powerful proponent for low-cost, self-run tests... and he's not, really, as I pointed out. Lukewarm at best, perhaps.
Meanwhile, why do you think he doesn't recommend testing for differences when he makes his announcements regarding findings? Or even mention that they vary?
This is just pure speculation, but it seems as if he's more concerned with sounding authoritative -- so people hire him -- than pointing out the ambiguities in statistical usability research.