▲ 354 points
back
107 comments
"Hub deliver content to each subscriber with a POST"
It's quite common on some social networks to have several thousands/millions subscribers for some pages/communities/accounts.
Is it really wise to build a publish-subscribe delivery system on top of HTTP? This seems to be a huge overhead.
In the meantime XMPP is already offering similar features (XEP-0060: Publish-Subscribe https://xmpp.org/extensions/xep-0060.html) for more than 10 years. It's implemented in several servers and can handle huge loads without problems (everything is handled in real-time through encrypted TCP sockets accross the network).
We are building social networks on top of XMPP for several years now, you can check Movim (https://movim.eu) and Salut à Toi (https://salut-a-toi.org/) :)
XMPP is an open TCP connection with its own protocol, like MQTT, Stomp, Redis, NATS, and all the *MQs. WebSub is mostly between HTTP servers, where one server can tell another that it wants a specific HTTP URL to be GET’d on a message. Each message requires a new TCP connection. So it will not handle very many messages - but I don’t think that’s the point.
There's no reason to open a new TCP connection for each HTTP request; multiple request per connection have been supported since HTTP 1.0, and since 1.1 it's even the default.
Is it really wise to build a publish-subscribe delivery system on top of HTTP?
I'm sure there is some use-case where this makes sense, but I agree with you. Probably most people wanting to do large scale pubsub should just be using XMPP, or possibly something like MQTT.
Azure Event Grid already provides "reliable event delivery at massive scale" using HTTP based pub sub: https://azure.microsoft.com/en-us/services/event-grid/
What's the overhead of HTTP vs XMPP? A few headers? Doesn't seem that different.
"a few headers" if 50% of your payload is headers, thats a big deal.
Mind you XMPP isn't all that efficient either. as its all based on XML.
XMPP uses decentralized architecture where communication is asynchronous. XMMP uses client-server model where client Do not talk directly to each other.
On the other hand HTTP is a simple protocol which is synchronous in nature.
> Is it really wise to build a publish-subscribe delivery system on top of HTTP? This seems to be a huge overhead.
I'm only just learning about WebSub tonight, but it looks like a lean, efficient, and fairly minimal protocol to me. What gives you the impression that there will be huge overhead - could you be more specific?
When new content is published to a topic in WebSub, it's delivered with an HTTP POST that will look something like this:
POST / HTTP/1.1
Host: foo.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 13
Link: <https://hub.example.com/>; rel="hub"
Link: <http://example.com/feed>; rel="self"
say=Hi&to=Mom
Furthermore, it looks like these messages can be sent using HTTP/2, if client & service support it (which is something that you'd prioritize for cases where efficiency matters). HTTP/2 is a binary protocol and takes advantage of HPACK header compression (RFC 7541). This means that if the same header appears in multiple requests, it will be transmitted very efficiently. Thus WebSub headers that are likely to be the same for all requests across a connection (like Host, Content-Type, and Link) will be transmitted virtually for free.
Even the vanilla HTTP/1.1 request described above seems reasonable though -- certainly not something that strikes me as a cost or efficiency problem -- and the HTTP/2 framing of the content is probably going to be not much longer than the content payload.
Now let's compare to XMPP PubSub. From looking at XEP-0060, an item published over that protocol looks like the following - based on Example 101 in: https://xmpp.org/extensions/xep-0060.html#publisher-publish
<message from='pubsub.shakespeare.lit'
to='francisco@denmark.lit' id='foo'>
<event xmlns='http://jabber.org/protocol/pubsub#event'>
<items node='princely_musings'>
<item id='ae890ac52d0df67ed7cfdf51b644e901'>
<entry xmlns='...'>
say hi to mom ...
</entry>
</item>
</items>
</event>
</message>
Based on this naive comparison, I don't see a reason to conclude that WebSub will have more overhead than XMPP PubSub. When implemented over HTTP/2 it may be more efficient.
Here you are comparing publishing an Atom post to XMPP and publishing a simple message. It will more look like something like that (even if it's not valid).
<iq type='set'
to='pubsub.shakespeare.lit'
id='publish1'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<publish node='mom'>
<item id='bnd81g37d61f49fgn581'>
<body>say hi to mom</body>
</item>
</publish>
</pubsub>
</iq>
That is also the power of Pubsub. It is that it gives you the freedom to put what you want in it (it can be Atom posts like in your example, but also stock market tickets that are pushed each 5 sec, some server monitoring logs...). You define your own namespace, write a little parser for it and use the thing into your XMPP Pubsub library :)
This is a proposal for a W3 standard. How many dependencies and upstream configurations do you need to realise XMPP in your data flow?
< "Is it really wise to build a publish-subscribe delivery system on top of HTTP?"
Seems to be working fine for SQS. It all depends on your use-case. For high volume messaging or certain types of messages you might reject WebSub for the same reasons you might reject SQS in favour of AMQP or MQTT etc.
SQS isn't really pub-sub. That'd be more Kinesis.
HTTP/2 presumably cuts out most of the overhead. Multiplexing sockets, header compression, etc.
Not to mention PubSubHubub...
This is the successor to PubSubHubbub. It’s pretty much the same API. https://websub.rocks is a great resource for implementors.
Why is this even supporting HTTP 1.1? And why HTTP at all and not just HTTPS? Encryption should be the default on all protocols going forward. No exceptions.
And I agree, it should use something like a Noise protocol even instead of HTTPS.
https://github.com/noiseprotocol/noise_spec/wiki/Noise-prope...
XMPP has already been mentioned (thanks edhelas). What I didn't see mentioned was ActivityPub (https://www.w3.org/TR/activitypub/), or some of the advanced messaging features users have come to expect (e.g., store and forward, forms, automated route handlers).
What I am curious about are the following questions?
1) What differentiates WebSub from XMPP?
2) What differentiates WebSub from ActivityPub?
3) How are you handling the N-squared delivery problem, if you are delivering content directly to each subscriber with HTTP POSTs?
4) Does WebSub currently support store and forward? If not, is that on a roadmap for a future version?
5) Same as 4, except for support for forms and form responses? Examples are a builtin Yes or No reply, or a builtin poll vote.
6) Same as 4, for automated message routing.
7) Why not have it be transport-agnostic, instead of mandating HTTP? And why HTTP? The growing trend is towards more decentralized.
8) How does this compare to Sir Tim Berners-Lee's SOLID (https://solid.mit.edu/)?
So I read the comments but I still don't understand what problem this specification solves. Does it enable new use cases? Does it enable new trust models? Since it's only a server protocol, how do final users actually read the content?
Edit : I don't understand the downvote. I use RSS daily to fetch news and I don't have problem with it. It's simple and deliver news to the edge (my mobile phone). I'm not saying WebSub is useless, I would like to understand what a 3 entities model brings to the table compared to a simple server-client delivery. What's more, the Subscriber entity cannot be a mobile device with the current network, because mobile internet providers block incoming GET requests. Therefore to fetch news, it has to be a pull model.
Why not add a simple rational in the header of the spec, explaining the problem, the existing solutions, and why this new solution? Does it solve a security issue, a scalability issue or a trust issue?
WebSub is the successor to PubSubHubBub, which was designed to make distribution of RSS items more efficient (ie receive items as they are published on the host):
- publisher would send updates, instead of having everyone poll - publisher would be protected from thundering herd if a content suddenly becomes popular - publisher and subscribers wouldn't need to exchange a full "page" of items when only one is needed
WebSub enables real-time web (push instead of pull). Like WebSockets, it's a protocol, not a technology. It's designed to be used in creating new technologies. One such example: https://superfeedr.com/tracker
If you subscribe to 1000 RSS feeds, it could take a while to update your feeds. Whereas with pubsub it's like all your feeds are merged and you can get a full update in one transaction. It's way more efficient at scale (I.e. when feeds are actual people, like in social networks)
1. Why not a SUB HTTP request? And a PUB http request. The response URL could be a required header.
2. We have HEAD, can we do service discovery using HEAD?
3. Why not let a topic be a HTTP URL? “PUB /user/john/position HTTP/1.1\r\ndata...”.
4. Subscription expiration as a way to force subscribers to renew and upon renew get redirected to other servers is pretty cool. NATS has a special message (the INFO message) to do the same, but you might be in the middle of an important request-reply session you don’t want to abort.
5. The authors could have made this protocol very “non-http-ish” by implementing what amounts to Redis but in HTTP. I’m glad they didn’t. This still feels like HTTP, which is great.
"topics" are indeed HTTP URLs:
1. Definitions
Topic. An HTTP (or HTTPS) resource URL.
Service discovery does appear to support HEAD requests. (See section 4.)
Having a new HTTP verb for subscribing and publishing would seem like unnecessary complexity to me. Rather than ask "why not a new verb", I think a case would need to be made that a new verb is required, that the operation does not cleanly fit into the semantics of existing verbs. The existing verbs are capable of modeling quite a lot.
With the protocol as they've described it, subscribing is just sending an HTTP POST to the hub URL, passing in the topic URL. That's a simple HTTP operation that a lot of clients and programs can be instructed to do easily. Requiring the use of a new HTTP verb will make interoperability difficult without apparent benefit.
I don't think we need more HTTP methods, unless a real case can be made for the existing ones not being good enough. The HTTP strongly discourages it.
But I love the idea of topics being Uris (not just HTTP)
This reminds me a lot of the SUBSCRIBE method [0] in Microsoft’s WebDAV extensions (I think this is used for EAS?). You pass a Call-Back header with a URL that gets called with NOTIFY [1]
[0] https://docs.microsoft.com/en-us/previous-versions/office/de... [1] https://docs.microsoft.com/en-us/previous-versions/office/de...
Middleboxes and bad HTTP servers/clients would explode, I guarantee it.
Superfeedr[1] (acquired by Medium in 2016[2]) was probably the biggest company yet, that based its entire business model on WebSub / PubSubHubbub protocol.
Just thinking beyond a replacement for RSS, this could be used for large-scale persistent virtual worlds. WebXR version of Second Life. GLTF as a content transmission format. As the Hub updates with a POST request, Subscribers callback could return local state. Hub becomes global state of truth. And there is no need to manage peer net or continuous WebSocket connections.
This has come along way from "WebSub was previously known as PubSubHubbub"
The former is a much better name IMHO!
So it's an alternative to something like WAMP? (Web Application Messaging Protocol on top of websockets - https://wamp-proto.org/)
Except you don’t have an open connection where you receive messages on. Instead, you register a callback url. It’s more like webhooks, but it’s it’s own thing, and not a setting on a page.
Just bring back Fidonet
How does this compare to ActivityPub used by Mastodon et al?
It's part of the Social Group at W3C IIRC. WebSub solves problems around RSS, notably that clients have to poll a lot and smaller sites thusly need to maintain those requests.
WebSub solves that by designating a hub that can handle this instead. Ie, you federate your blog feed to elsewhere.
ActivityPub fills another niche, mostly everything around human interaction in social networks.
WebSub could be used to feed data into ActivityPub networks.
After reading the responses I don't see yet why the non-transport bits of WebSub can't be done with ActivityStreams and ActivityPub. That said, I haven't put the kind of time into it like the W3C Social Group has, nor am I recognized expert on it, so I'll hold further comment until I can finish and publish to Show HN a related proof-of-concept involving Peer-to-Peer topic based publish/subscribe using ActivityPub, that I've had on the back-burner for a while.
See it more as an alternative to RSS. Aggregators subscribe and feed information to their users. The polling step is just switched to pushing.
I wish there was a pull aspect to it, like NEWNEWS with NNTP. If the subscriber is offline it seems that there doesn't seem to be a way to retrieve missing items except through the publisher.
> The subscriber must be directly network-accessible and is identified by its Subscriber Callback URL
Does this mean the subscriber needs to have a forwarded port open to the internet for this to work? Without IPv6, users behind NAT (and specifically behind CGNAT) wouldn't be able to use it.
I think Websub is enhancement to XMPP. Both are asynchronous. Which means it should be faster than HTTP. Actually it's a push-based protocol where server sends single POST to the hub, the hub then sends that notification to all subscribers.
So as with webhooks, this will require you to have a permanently reachable server sitting on the internet. Couldn't they at least have defined an alternative transport using websockets or SSE?
WebSub is essentially just the transport; the payload is usually Atom or RSS, but it's not specified. Had they included a websockets version, it would essentially be something almost completely different with the same name.
What I don't get is what the motivation is for running the hub, other than "get bought by Medium?"
Is this compatible with new distributed crypto projects like scuttlebutt or dat, and if not, why not?
WebSub is just the standardization of PubSubhubbub, which has existed (and been used by publishers like Feedburner) for longer than those projects.
It is not, because it predates them both by several years.