Does that even test for intelligence?
This is like testing if a horse can fly just because someone showed an image of a Pegasus, or testing if a fish can climb up a tree and believing they are not intelligent because each of them cannot fly or climb up trees.
Pulling things into the ridiculous isn't condusive to a good faith discussion and does not help proving pseudoscientificness which you seem to be after. (I don't belive that the pelican bicycle test is meant to be a serious scientific endeavour, btw.)
Then use a model that supports multi-modal output then, instead of using models that only support raw text output to construct an image.
> If it can't even do that well then I'm not sure why we need to talk about horses here.
Testing a model that only outputs text and forcing it to output an unsupported modality is obviously not going to work well which is what it going on here.
Would you expect Claude/GPT to replace your compiler because it can output text? That would be like expecting that horses can fly because someone saw an ancient winged-horse on a brick tablet.
> Pulling things into the ridiculous isn't condusive to a good faith discussion and does not help proving pseudoscientificness which you seem to be after.
That is because the premise of the test is ridiculous and pseudoscientific.
> (I don't belive that the pelican bicycle test is meant to be a serious scientific endeavour, btw.)
Why did you say it was a "test for intelligence", when we both know it is a joke that tests for nothing?
Somebody who has trained painting animals and vehicles for decades is surely skilled at that, but is not solving new problems, so is not necessarily applying intelligence when painting something of that sort.
You can take a reasonably skilled programmer, show them some C code and ask them to write equivalent ASM code, perhaps providing them with a language reference for either, and they will be able to do this translation. Slowly, but they will get there. They have not been trained to do this, but they will "figure it out". With enough time and motivation, even non-programmers would be able to do that. That is what intelligence is and of course Claude/GPT needs to be able to do that if it wants to claim that it intelligent. No flying horses.