Deno Queues

The most interesting detail is probably the schema they're using for that:

    CREATE TABLE queue (
      ts integer not null,
      id text not null,
      data blob not null,
      backoff_schedule text not null,
      keys_if_undelivered blob not null,
      primary key (ts, id)
    );
    CREATE TABLE queue_running(
      deadline integer not null,
      id text not null,
      data blob not null,
      backoff_schedule text not null,
      keys_if_undelivered blob not null,
      primary key (deadline, id)
    );
    CREATE INDEX kv_expiration_ms_idx on kv (expiration_ms);

I'm using Go for the first time this year and some aspects of the language are very C like. Of the list of things that isn't C like (e.g. GC allowing one to return what look like stack allocated pointers from functions) there is the obvious inclusion of things like `map[string]string`. I bring this up because it struck me that inventing a language in 2023+, it would seem almost insane not to have built in syntax for the language to handle map types.

And so it seems logical that a web-server focused eco-system would start to garner libraries and even language syntax (maybe one day) for the kind of primitives we frequently use on web-servers. I mean, I can't even recall the last time I worked on a distributed web server that didn't have a KV store as a cache, or a locking mechanism, or even an ad-hoc queue. A distributed system without a KV store feels vaguely isomorphic to a computer language without a map type. It is such a Swiss army knife kind of technology.

One potential issue is that Deno is going alone down this path. Currently, I don't feel confident that the new features they are adding will be available on competing platforms. Even if they open source the API (it is on a `Deno` namespace), I'm not sure it will just work on AWS if I wanted to switch out FoundationDB for e.g. Redis.

For that reason, I feel I want to avoid Deno. Even if the syntax and exposed features are really attractive, I'm worried about becoming locked in and then having to do a lot of surgery to code to make it deployable on multiple cloud infrastructures. That is a requirement from a lot of clients. E.g. maybe I want to sell to Oracle or Salesforce one day but they mandate I have to run my systems on-prem. I'm now on the hook trying to figure out how to adapt whatever available KV store they have to the features I'm using from the `Deno` package.

It is a double edged sword. Maybe they will succeed in pushing forward this vision to a broader audience. For now I'll probably remain cautious.

I don't buy this line of reasoning. At the end of the day we are building infrastructure that needs to be reliable. Spending 30 minutes to set up an SQS queue (proven technology) doesn't sound as bad as putting all my trust in a toy queue that Deno built "on top of SQLite / FoundationDB". Is the 30 minute setup cost and added "developer experience" really worth the risk? How often are you setting up queues anyway.

> user code.

A past life has taught me that users will never properly understand at-least-once semantics. Anytime you redeliver messages, you will get a flurry of user complaints and breakage.

Either you do the impossible and invent a way to do exactly-once semantics. Or you should always redeliver 0.1% of all messages, just so that users don't come to depend on messages being delivered once.

For the local version you could get multiple queues by calling Deno.openKv("db-2.db") with different SQLite file paths each time, but that feels like a lot of overhead for a pretty common need.

I guess this is a Deno architectural style thing - maybe when you build complex apps on Deno it's expected that you'll have a microservice style architecture where lots of different scripts work together, each of them with their own KV store and hence their own queue?

I thought Deno was some sort of node.js replacement. What am I missing and can I use this either locally and/or self hosted without paying for it?

But I won't write servers in Deno and put them to production in my own servers. (I don't like serverless)

You see, Deno(and Bun)'s business model is built around serverless (Deno deploy). And Deno deploy is ANOTHER runtime.

It's clear with KV and queues... Locally and if you self host, you have a barely working version backed by another tech...

This means you're self hosting a version of Deno that's different from the majority of the Deno customer base.

How many people will use Deno to run servers on their own hardware? Since that number is low, then I'd rather use Node.

[1]: https://docs.deno.com/deploy/api/runtime-broadcast-channel

In case anyone wonders what this would make possible out of the box: https://apple.github.io/foundationdb/design-recipes.html

It does show a level of understanding by the language creator that they know how the language can and probably should be used - I like how KV defaults to SQLite locally and takes on new meaning in a hosted environment. I haven't seen, but as long as you can actually override the behavior to use your own KV/queue technology (so long as it satisfies the interface) I see absolutely no issue with this.

There is a huge benefit where the code always looks the same for common things we all need to do. I wouldn't mind seeing other languages take an overall stab at this so there is some choice other than JavaScript.

Some questions:

* Looks very powerful, but at the core is it just one single queue? Does Deno.openKv() create a new queue or re-use the same queue every time it's called?

* Usually I need multiple queues of different priority, so when bottlenecks happen the highest priority jobs run first.

* Maybe they guarantee infinite worker capacity so you don't need queue priorities? No bottlenecks if you can pay for the jobs you enqueue?

[1] https://www.google.com/search?q=Use+database+as+a+queue+is+h...

A bit of an aside, but not sure we want at-most-once delivery for email.

  Enqueuing a Message: Each enqueue action translates into a KV write operation.

  Receiving a Message: Every received message entails a KV write, and a single request charge.

I really like this

    With grand promises rode the ploy
    Venture capital, ahoy!
    We make a lovable product to capture audience
    Vendor lock them at the earliest convenience
    Hey look at our awesome developer experience
    Add ANSI and emojis and comments saying it’s genius
    Some next level tech, plebs just don’t get it
    Next round of funding, we’ve already spent it
    I took the big dough, got puppets and framed it
    R-O-I is bad though but layoffs gon’ fix it
    Never mind the drama *cough* are ya gon’ install it
    Run it, ship it, kill yourself with working on it
    Entangle your business with it
    Tell the client to deal with it
    Bring it to the big wig
    Get ’em to sign off on it
    Crown your own sh*t
    Dogfood is tasty, ain’t it

I understand its only when using their cloud and there is a local implementation, but I just couldn't imagine using a feature in LLVM, for example, where if you were to upload the binary to a specific cloud it would use a different implmentation that costs behind the scenes.

It just feels odd to me. I guess it's kind of cool. Maybe different clouds can implement the underlying KV and then this becomes cross-cloud with different underlying benefits without changing the code. But I'm not entirely sure how I feel about this.

Is there any full stack solution other than Fresh?

edit: people seem to be getting hung up on the "runtime" v "programming language" distinction. I'm not sure why--it's weird to me that this is "part of the language + runtime" at all. Clearly, reasonable people can disagree about that, but hopefully on points more substantive than pedantic

If tomorrow, someone started a for-profit company making a faster Python runtime and introduced features like this, it would be weird, and it would feel as pointless to me as Deno is