> To test that progress, we returned to a task the old swarm had struggled with: building SQLite from scratch, in Rust, from nothing but its documentation.
Isn't SQLite's source code in its training data?
back
2 comments
Of course it is
not in Rust. might be different enough ...
Even if no Rust code for it was seen during training, an LLM can trivially transpile SQLite's C codebase to Rust on the fly. For example, I just asked ChatGPT to write John Carmack's famous Fast Inverse Square Root algorithm in Erlang, without searching online or thinking, and it transpiled it immediately (while also extracting the knowledge in the same step). SQLite's semantics/code are stored in the middle layers of an LLM, and the last layers are able to convert it into any representation, as conditioned by the prompt. Cursor's experiment is deeply flawed because they merely extracted the model's compressed, lossy knowledge of SQLite's codebase and then just ran a bunch of tests/fixing rounds to make up for the lossiness. The claim that the agents built it from scratch is false.
Then you would expect the implementation to be structured the same as SQLite, and having glanced at the result, it looks like at least some things aren't. For example, it seems to use an operator-tree executor rather than SQLite's bytecode interpreter.
The knowledge is lossy, and code generation itself is non-deterministic (temperature), so the operator-tree executor must be interference from other DB implementations, because it's uncommon to have a bytecode interpreter
The point is that the model has the source code in its weights. Therefore it's misleading to focus on being "only given the documentation", when it could produce a SQLite implementation in rust with no documentation at all. The wording is designed to create a marketing impression that it can do spec-driven development for a novel project, but this example doesn't actually support that.
It's like a diet system advertising how much weight a spokesperson lost on their system, without disclosing that they also went on Ozempic at the same time.
To test this hypothesis you would need to look at the initial version before subsequent agents started refactoring it.
Why tho? The whole point of agentic is that it takes several passes to get to a result. If step a) is a non-idiomatic transpile, but step b) c) and ... z) make it idiomatic and performant and efficient, why look at step a)? That was just the beginning. The idea with "agents" and "swarms" and dynamic this and that is that you only look at the last step.
This strictly follows the early agentic usage, where people were complaining of code full of errors and such. But having the loop do plan-write-verify is the thing that got us from that state to today. And it works. If you look at the code and it has errors (linting, etc) then your loop is not properly setup. You should only be called upon when all that can and should pass does. Your role is "direction/taste/human-common-sense-at-a-glance".
Because the point of the test is to see what the agentic swarm approach is capable of. If your test assumes that a version of the thing you are trying to build is available in the training data, or can't determine if it matters whether the thing you're building is in the training data, then it's not a very useful test. Most of the time when you consider agentic approaches, you're not trying to rebuild something the agents involved are trained on.
I see, that makes sense. Thank you.
There's Turso which is a sqlite rewrite in Rust with some additional features. It's open source so I'd imagine it would be in the training data.