Item — HN Reader

Great intro to PDB, but I have to be critical of: "The second group took 9 minutes on average, and the first one 14 minutes! Those are not completely statistically significant, but that’s still a 35% speedup on an 8-line-long program."

If you want to talk about statistical significance, don't say "not completely". It either is or it isn't. If you think the p-value is important (and I'm not saying it is, but don't bother mentioning it if you don't think it is), then you can't claim a 35% speedup. If you're reporting results in the context of significance, and you decided that your results were not significant, then the 35% speedup is meaningless.

1 comments

Article author here. I agree with your point, I shouldn't have been so cautious. I consistently observed people being faster when using debuggers and I wanted to give it a test on this problem. Most people don't see that they can get a significant speedup in using debuggers, so that sentence wanted to give a sense on how much quicker you can be. I'd be happy to hear of a better way to convince a non-debugger user that he could have debugged this code faster with a debugger!

Well I was convinced :) If it were me, I'd probably just remove the mention of significance -- or if you have more people and some time you can probably set up a proper experiment, control for some confounds and you'll probably get significant results.