back
22 comments
Be sure to check out http://sqlite.org/src4/doc/trunk/www/design.wiki ... SQLite4 splits the SQL engine from the storage engine and provides public APIs for both, including the ability to run the SQL engine on top of any key/value store that provides an ordered-map interface.

In the new design a row's primary key directly contributes to its physical key, meaning lookups and iterations on the primary key involve only one scan (rather than an index scan followed by random lookups in SQLite 3.x/most DBMS). Basically the old "OID" feature on steroids, except now you don't have to design for it at all. This also means data is inherently clustered by primary key, providing further performance benefits.

The new storage engine in 4.x is exciting by itself, it's a log structured merge tree, similar to LevelDB but using the old SQLite 3.x B+tree instead of SSTables for the disk segments. Additionally it is possible to incrementally compact older generations inline as part of each write, rather than mandatorily having some unpredictable background compactor thread like LevelDB. Similar to 3.x and unlike LevelDB, the storage engine also supports interprocess locking, meaning it's trivial to write software with concurrent readers across multiple processes and a single writer.

The possibility of mixing a key/value store and SQL store, including having single transactions affecting both stores, is really exciting. The SQL engine is still a work in progress, but it should be awesome when it's ready.

"In the new design a row's primary key directly contributes to its physical key, meaning lookups and iterations on the primary key involve only one scan (rather than an index scan followed by random lookups in SQLite 3.x/most DBMS)."

Isn't this basically the same concept as a clustered index in MS SQL Server?

Yep, although I'm not sure if clustering is also a hand-triggered process in SQL Server (like it is in e.g. PostgreSQL). In any case SQLite 4.x will do it automatically.
In SQL Server that's defined at table creation time.

In Oracle it's the same: defined at creation time, but you call it an Indexed Ordered Table (IOT).

In MySQL all InnoDB tables are clustered (in that sense) because that's the only storage strategy of InnoDB.

> ... including the ability to run the SQL engine on top of any key/value store ...

Note that you can do that today with the virtual tables interface in SQLite 3. For example several people have written virtual tables where the underlying data is in a csv file, I wrote one that backends into CouchDB etc. It is a little ugly in that SQLite semantics bleed through the interface (eg BestIndex method) but does work. http://www.sqlite.org/vtab.html

Those who are interested in SQLite4's LSM and LevelDB might also find OpenLDAP's MDB: http://symas.com/mdb/ interesting.
Beware MDB isn't quite as general purpose as the other options, specifically feeding it large records can cause it fall back to placing records in pages of their own, which can be wasteful on storage. It's easy to work around (by splitting your records up) but this needs to be considered as a factor of your design.
Ha ha. The foreign_key_check command is going to be a lifesaver for me.

I run with foreign key support disabled, because the referencing ("child") rows might get inserted before the referencing ("parent") rows (I don't control the order of inserts).

Now I can just run a periodic scrub job with this command to clean out rows which didn't successfully transition from "violating foreign keys" to "satisfying foreign keys", instead of writing my own code.

Neat. SQLite is an awesome database library.

Coming from the FoxPro days, I'm a huge fan of SQLite.
I didn't imagine I'd ever see FoxPro mentioned on Hacker News. That's like seeing... a Gnome vs. KDE debate on a FoxPro forum.
I had an exam in Ashton Tate's dBase in the last year at my high school (this was a special mathematical high school in Bulgaria, every big city had one such school or more)
What about Clipper and dB3?
Started my career on dBase II, then III, then Paradox, then Foxpro (and VFP). Eventually on to DB2/2 (OS/2), Sybase, MSSQL, now all PostgreSQL.

The FoxPro / xBase market was very nice to me for about 6 years. I am happily surprised to see this still exists: http://www.dbase.com/

At my first job, I had a foxbase database (government built), Turbo C 2.01 (make sure to patch the FP), and a Postscript printer. We needed some reports and graphs to satisfy other agencies, what fun.
No idea what's FoxPro, but I'm huge fan of SQLite as well.

The only complain (not sure to whom) I have is that WebSQL have not got proper attention because SQLite is the only solution.

http://en.wikipedia.org/wiki/FoxPro

http://en.wikipedia.org/wiki/Visual_FoxPro

It had a lot of fans back in the day.

Thanks for the blast from the past! FoxPro was leading edge in its time. I'm also a fan of SQLite; it works great for my website. Who needs bigger iron (on a single machine anyway) when SQLite can insert a million+ rows a second?
FoxPro 2.0/DOS was really disruptive product back then. And it was addictive. My academic advisor used to using FoxPro as his primary text editor.

But real guys preferred Clipper due the advanced abilities such as eval.

I used to use it as a local database for desktop (and CD-ROM based) Windows apps I created with Flash, wrapped in a SWFStudio wrapper. :)
I am curious... why do you find sqlite similar to FoxPro? I thought the inner engines were quite different
Actually, rather similar. Visual FoxPro used SQL syntax for large majority of it's database and free table operations. Some commands were slightly different, and allowed you to use VFP specific commands in them. But, transitioning a VFP app to use a SQL backend was relatively simple.

Actually, my shop still runs a few VFP tasks that connect to various MySQL backends as well as JSON API services.

Side note, there is a very useful Python library for working with VFP xBase (.dbf) tables: https://pypi.python.org/pypi/dbf

I thought you meant real FoxPro - like 2.x for DOS/Unix.

Now, that was a desktop database development system. Lightning fast, portable between SCO and DOS, easy to work with. Made clipper and dBase look like pokey hunks of poop in comparison. That and netware 3.12, plus some low powered machines with token ring cards for clients - golden days, I tell you.