Item — HN Reader

I love this type of research, but this particular study requires a two-ton grain of salt.

First, these results are based on only 20 total homicide offenders. So when they claim they can classify 75% of murderers correctly, that really just means 15 out of 20.

Second, the final classifier has access to more than 14 variables, including factors like the number of previous convictions, not just brain structure. [1]

If you're fitting 14 variables to 20 data points, it's very likely you're over-fitting. I personally would wait for a much larger study before drawing conclusions.

[1] "The identified variables were age, PCL:YV Factor 1, PCL:YV Factor 2, years of regular substance use, total number of convictions, ICU, SES, brain volume, left and right lateral orbital frontal cortex, medial orbital frontal cortex, anterior and posterior cingulate, right temporal pole, and right and left parahippocampal cortex." http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055901/

3 comments

20 total homicide offenders, 135 non-homicide offenders, and two control groups comprising another 41 subjects between them. I agree it's a highly preliminary result, but successfully IDing 75% of homicide perpetrators and 82% of non-homicide subjects is a good start. Why are you leaving out the rest of the test population?

PS I don't think it's perfect by any means, when you look at the population it's pretty particular to New Mexico. Per my other comment, I think the significance of this is that we should be doing a lot more research in our depressingly large incarcerated population to develop a better understanding what makes some criminals dangerous vs others who are just selfish or opportunistic.

I think the key numbers to compare in a study like this are the number of positive labels (in this case, homicide offenders) vs. the number of variables in the model. With n positive labels and n variables, you can almost always find a set of weights that achieve high accuracy. (That applies even though they're using cross-validation -- I suspect the experimenter tried multiple treatments, so it's very easy to trick oneself into believing you've found "the" model.)

For the control set, the model gets 82% accuracy, vs 87% if they had simply classified everybody as a non-murderer. That suggests mild over-fitting to me.

The other two controls are interesting, especially the one trying to match demographic variables, but the sample sizes (n=20, n=21) are too small IMO to conclude much, and I don't think you can pool the control groups since they're drawn from different populations.

That's not to say their conclusions are wrong. But studies on small sample sizes produce spurious results all the time.

The flip side of that observation is that large-scale machine learning techniques applied to relatively "shallow" data sets might produce some very surprising results.

75% of homicide perpetrators identified correctly (15/20) + 82% of non-homicide perpetrators identified correctly (111/135) is a total classification accuracy of 81% (126/155) for the procedure.

But if we just assume that everyone is in the non-homicide group (which is what "innocent until proven guilty" has us do anyway), then that gives us a total classification accuracy of (135/155) 85% for the same sample set. That is, "innocent until proven guilty" is notably more accurate than the system which is being proposed here, and infinitely less likely to deliver a false positive result.

It can't be infinitely less likely to produce a false result or we'd never have miscarriages of justice. Current research estimates the rate of false positives in capital crimes (almost all of which are homicides) at 4.1%. http://www.pnas.org/content/early/2014/04/23/1306417111.full...

But the point here is that chance is only 50%, so if we have a mechanical system that's within shouting distance of matching our extremely elaborate and very expensive trial system then that's noteworthy. 81% is not far away from 87%; what happens when the machine learning system can repeatably identify homicidal behavior with greater accuracy than our criminal justice system? Overall accuracy is highly significant because it will create a lot of pressure to identify murderers in the interests of public safety. Of course we shouldn't rely on such a system because of the risk of false positives, but you'll see attempts to bring it into evidence in more and more cases. If/when such a system starts to outperform the existing on in terms of having a lower number of false positives, then every innocent defendant is going to demand it and every guilty defendant (in the legal sense) will cite it as a mitigating circumstance.

I love commenting on neuroimage research after reading the fantastic:

"Ten ironic rules for non-statistical reviewers." Neuroimage 61.4 http://www.fil.ion.ucl.ac.uk/~karl/Ten%20ironic%20rules%20fo...

Also, correlation does not imply causation. For all they know incarceration of a developing mind causes grey matter differences.

No, they are comparing criminals incarcerated for murder w. criminals incarcerated for something else.

There might exist an (IMHO unlikely) argument that killing someone causes grey matter differences, but the effects of incarceration and such are controlled for.