▲ 230 points
back
104 comments
I would strongly implore people not to follow the example this post suggests, and write code that relies on this monotonicity.
The reason for this is simple: the documentation doesn't promise this property. Moreover, even if it did, the RFC for UUIDv7 doesn't promise this property. If you decide to depend on it, you're setting yourself up for a bad time when PostgreSQL decides to change their implementation strategy, or you move to a different database.
Further, the stated motivations for this, to slightly simplify testing code, are massively under-motivating. Saving a single line of code can hardly be said to be worth it, but even if it were, this is a problem far better solved by simply writing a function that will both generate the objects and sort them.
As a profession, I strongly feel we need to do a better job orienting ourselves to the reality that our code has a tendency to live for a long time, and we need to optimize not for "how quickly can I type it", but "what will this code cost over its lifetime".
> […] code that relies on this monotonicity. The reason for this is simple: the documentation doesn't promise this property. Moreover, even if it did, the RFC for UUIDv7 doesn't promise this property.
The "RFC for UUIDv7", RFC 9562, explicitly mentions monotonicity in §6.2 ("Monotonicity and Counters"):
Monotonicity (each subsequent value being greater than the last) is
the backbone of time-based sortable UUIDs. Normally, time-based UUIDs
from this document will be monotonic due to an embedded timestamp;
however, implementations can guarantee additional monotonicity via
the concepts covered in this section.
In the UUIDv7 definition (§5.7) it explicitly mentions the technique that Postgres employs for rand_a:
rand_a:
12 bits of pseudorandom data to provide uniqueness as per
Section 6.9 and/or optional constructs to guarantee additional
monotonicity as per Section 6.2. Occupies bits 52 through 63
(octets 6-7).
Note: "optional constructs to guarantee additional monotonicity". Pg makes use of that option.
>explicitly mentions monotonicity
>optional constructs
So it is explicitly mentioned in the RFC as optional, and Pg doesn't state that they guaranty that option. The point still stands, depending on optional behavior is a recipe for failure when the option is no longer taken.
The test should do a set comparison, not an ordered list comparison, if it wants to check that the same 5 accounts were returned by the API. I think it's as simple as that.
The blogpost is interesting and I appreciated learning the details of how the UUIDv7 implementation works.
Don’t you think that depends on what you’re guaranteeing in your api? If you’re guaranteeing that your api returns the accounts ordered you need to test for that. But I do agree in general that using a set is the correct move.
As a counter-argument, it will inevitably turn into a spec if it becomes widely-used enough.
What was that saying, like: “every behavior of software eventually becomes API”
In enterprise land. In proof of concept land that's not quite true (but does become true if the concept works)
I agree, optimizing for readability and maintainability is almost always the right choice.
> Moreover, even if it did, the RFC for UUIDv7 doesn't promise this property.
Huh?
If the docs were to guarantee it, they guarantee it. Why are you looking for everything to be part of RFC UUIDv7?
Failure of logic.
I too am missing the win on this. It is breaking the spec, and does not seem like it offers a significant advantage. In the eventual event where you have a collection of UUID7 you are only ever going to be able to rely on the millisecond precision anyway.
I don't think most people will heed this warning. I warned people in a programming forum that Python ordering of objects by insertion time was a implementation detail, because it's not guaranteed by any PEP [0]. I could literally write a PEP compliant Python interpreter and could blow up in someone's code because they rely on the CPython interpreter behavior.
[0]: https://mail.python.org/pipermail/python-dev/2017-December/1...
Most code does not live for a long time. Similar to how consumer products are built for planned obsolescence, code is also built with a specific lifespan in mind.
If you spend time making code bulletproof so it can run for like 100 years, you will have wasted a lot of effort for nothing when someone comes along and wipes it clean and replaces it with new code in 2 years. Requirements change, code changes, it’s the nature of business.
Remember any fool can build a bridge that stands, it takes an engineer to make a bridge that barely stands.
Remember even if timestamps may be generated using a monotonically increasing value that does not mean they were committed in the same order to the database. It is an entirely separate problem if you are trying to actually determine what rows are "new" versus "previously seen" for things like cursor-based APIs and background job processing. This problem exists even with things like a serial/autoincrement primary key.
+1
What could be useful here is if postgres provided a way to determine the latest frozen uuid. This could be a few ms behind the last committed uuid but should guarantee that no new rows will land before the frozen uuid. Then we can use a single cursor track previously seen.
>The Postgres patch solves the problem by repurposing 12 bits of the UUID’s random component to increase the precision of the timestamp down to nanosecond granularity [...]
>It makes a repeated UUID between processes more likely, but there’s still 62 bits of randomness left to make use of, so collisions remain vastly unlikely.
Does it? Even though the number of random bits has decreased, the time interval to create such a duplicate has also decreased, namely to an interval of one nanosecond.
I could imagine that certain nanoseconds might be vastly more likely than other nanoseconds.
For example, imagine you have a router that sends network packets out at the start of each microsecond, synced to wall time.
Or the OS scheduler always wakes processes up on a millisecond timer tick or some polling loop.
Now, when those packets are received by a postgres server and processed, the time to do that is probably fairly consistent - meaning that X nanoseconds past the microsecond you probably get most records being created.
Imagine if you were generating 16 UUIDs per nanosecond, every nanosecond.
According to [1] due to the birthday paradox, the probability of a collision in any given nanosecond would be 3E−17 which of course sounds pretty low
But there are 3.154e+16 nanoseconds in a year - and if you get out your high-precision calculator, it'll tell you there's a 61.41% chance of a collision in a year.
Of course you might very well say "Who needs 16 UUIDs per nanosecond anyway?"
Depends if you think sub-millisecond locality is significant.
I maintain that people are too eager to use UUIDv7 to begin with. It's a dessert topping and a floor wax.
Let's say you need an opaque unique handle, and a timestamp, and a monotonically increasing row ID. Common enough. Do they have to be the same thing? Should they be the same thing? Because to me that sounds like three things: an autoincrementing primary key, a UUIDv4, and a nanosecond timestamp.
Is it always ok that the 'opaque' unique ID isn't opaque at all, that it's carrying around a timestamp? Will that allow correlating things which maybe you didn't want hostiles to correlate? Are you 100% sure that you'll never want, or need, to re-timestamp data without changing its global ID?
Maybe you do need these things unnormalized and conflated. Do you though? At least ask the question.
You can keep all three and still use UUIDv7 as a performance improvement in certain contexts due to data locality.
Also if you have a nanosecond timestamp, do you actually need a monotonically increasing row ID? What for?
Re-timestamp would be a new one for me. What’s a conceivable use case? An NTP fault?
> The Postgres patch solves the problem by repurposing 12 bits of the UUID’s random component to increase the precision of the timestamp down to nanosecond granularity (filling rand_a above), which in practice is too precise to contain two UUIDv7s generated in the same process.
A millisecond divided by 4096 is not a nanosecond. It's about 250 nanoseconds.
UUID7 is excellent.
I want to share a django library I wrote a little while back which allows for prefixed identity fields, in the same style as Stripe's ID fields (obj_XXXXXXXXX):
https://github.com/jleclanche/django-prefixed-identity-field...
This gives a PrefixedIdentityField(prefix="obj_"), which is backed by uuid7 and base58. In the database, the IDs are stored as UUIDs, which makes them an efficient field -- they are transformed into prefixed IDs when coming out of the database, which makes them perfect for APIs.
(I know, no documentation .. if someone wants to use this, feel free to file issues to ask questions, I'd love to help)
My org has been using ULID[0] extensively for a few years, and generally we've been quite happy with it. After initially dabbing with a few implementations, I reimplemented the spec in Kotlin, and this has been working out quite well for us. We will open-source our implementation in the following weeks.
ULID does specifically require generated IDs to be monotonically increasing as opposed to what the RFC for UUIDv7 states, which is a big deal IMHO.
Having used a lot of the ULID variants that the UUIDv7 spec cites as prior art, including the ULID spec you link to, I've gotta say that UUIDv7 has some real advantages.
The biggest advantage is that it is hex. Haven't yet met a database system that doesn't have functions for substr and from_hex etc, meaning you can extract the time part using vanilla sql.
ULID and others that use custom variants of base32 or base62 or whatever are just about impossible to wrangle with normal tooling.
Your future selfs will thank you for being able to manipulate it in whatever database you use in the future to analyse old logs or import whatever data you generate today.
ULID guarantees monotonicity only per process, and it requires ID generation to be serialized. I find the promise quite misleading because of that. You might as well use a wide-enough integer with the current timestamp + random as baseline for the same purpose, but I wouldn't recommend that either.
What benefit does this have over something like Twitter's Snowflake, which can be used to generate distributed monotonically increasing IDs without synchronization?
We've been using an implementation of it in Go for many years in production without issues.
UUIDv7 interoperates with all the other versions of UUID. The v7 support in Postgres doesn't add a new column type, it makes the existing column type more powerful/capable. Applications that had been using UUIDv4 everywhere can get cheap Snowflake-like benefits in existing code just from switching the generator function. Most languages have a GUID or UUID class/struct that is compatibly upgradable from v4 to v7, too.
Ordering for UUIDv7s in the same millisecond is super useful when some rows represent actions and others reactions.
I have used this guarantee for events generated on clients. It really simplifies a lot of reasoning.
This post makes me think (keep thinking) if we can use a solution that I used for another project in another context: using a Cryptographic Feistel Network to compute UUIDS so they are reversible if you need to know the original order. Each entity uses another key for the generation but if they know the keys they can know the order of the other party. Basically is using an existing cryptographic function if the block size is the same and if not adaping it to a specific block size via a Feistel Network.
On one hand I too am looking forward to more widespread use of UUIDv7, but on the other I don't really get the problem this is solving for their spec. If you care about timestamp ordering I don't think doing it in a way that forces you to fake a PK if you insert an earlier dated record at a future point makes sense. But I guess I'm implicitly assuming that human meaningful dates differ from insertion times in many domains.
After reading this I went ahead and added the extra 12 bits to my Go UUID library. Thanks for the write up on the PostgreSQL patch.
If anyone is interested, here is the package: https://github.com/cmackenzie1/go-uuid. It also includes a CLI similar to that of `uuidgen`, but supports version 7.
I have an implementation function that computes N v7 UUIDs, sorts them, and returns them. This makes testing possible.
Collection<UUID> generate(final int count);
The naming of "rand_a" and "rand_b" in the spec is a bit misleading. They don't have to be generated randomly. I'm sure there's a historical reason for it.
"extra_" or "distinct_" would be a more accurate prefix for UUIDv7.
UUIDv7 is actually quite a flexible standard due to these two underspecified fields. I'm glad Postgres took advantage of that!
I implemented this in pure Python a few days ago in case anyone finds it helpful, here it is: https://gist.github.com/pirate/7e44387c12f434a77072d50c52a3d...
My implementation supports graceful degradation between nanosecond scale resolution, microsecond, and millisecond, by using 12 bits for each and filling up the leftmost bits of rand_a and rand_b. Not all environments provide high resolution system clocks with no drift, so it's is important to maintain monotonicity when generating IDs with a low-res timestamp as input. You still want the bits that would've held the nanosecond value to be monotonic.
Neither of the existing uuid_utils and uuid7 python libs that can generate UUID7s support this monotonicity property.
Am planning on using this for ArchiveBox append-only "snapshot" records, which are intrinsically linked to time, so it's a good use-case imo.
There's another great resource here that I think is one of the best explainers of UUIDv7: https://antonz.org/uuidv7/
Whatever you do, don't implement the cursed 36-bit whole-second based time UUIDv7 variant that you occasionally see on StackOverflow / blog posts, stick to 48!
This looks great, thanks. But I think gists are better for unimportant stuff - for this I think it deserves its own repo.
Been waiting for UUIDv7 for years -- maybe it's time to archive pg_idkit[0], or maybe instead just switch the UUIDv7 version to do the native thing rather than the Rust code.
What is a v7 UUID. Why do we need more than 1. uuid from a random seed and 2. one derived from that and a timestamp (orderable)
UUID v7 is what you numbered #2.
For the others, it’s best to read up on Wikipedia[0]. I believe they all have their unique use-cases and tradeoffs.
E.g. including information about which node of the system generated an ID.
[0]: https://en.m.wikipedia.org/wiki/Universally_unique_identifie...
As it is usual in many areas of human endeavor, newcomers to the field tend to criticize design decisions that were made before them, only to re-invent what was already invented.
Sometimes it leads to improvements in the field, via rejection of the accumulated legacy crud, or just simply affording a new perspective. Most other times it's a well-intentioned, but low-effort noise.
I, personally, do it myself. This is how I learn.
UUID v7 is the latter, whereas v4 is the former.
All the other versions are somewhat legacy, and you shouldn't use them in new systems (besides v8, which is "custom format UUID", if you need that.)
A deterministic uuid based off of a hash of bits is also very useful (UUID5). I've used that for deduping records from multiple sources