ARC-AGI is not yet killed by LLM. O3 achieved a breakthrough only on ARC-AGI-PUB, which is semi-private. Nothing guarantees that the test data wasn't leaked to OpenAI in previous testing rounds, because the model is not running offline.
back
2 comments
I think this should be discussed more. Models that can only be accessed via API cannot be tested without giving their owners access to the test data. You just have to trust that they’ll do the right thing.
In particular, in cases where the model gets 16 hours to solve a task that a human can solve in a few minutes, cheating is trivial!
See https://arcprize.org/blog/oai-o3-pub-breakthrough
ARC-AGI-1 will be replaced by ARC-AGI-2
So yes, ARC-AGI-1 was killed.
ARC-AGI-2 was planned long before those results came out. Also from the link: ARC-AGI-2 (same format - verified easy for humans, harder for AI) will launch alongside ARC Prize 2025. So, no, it will not replace it.
Was it? I think reaching >70% made it a necessity.