back

by softwaredoug·7y ago·view on hn ↗
Just want to mention, WMF has a very small but elite team of engineers. Amazed they maintain an Alexa top 5 site with many orders of magnitude less engineering staff than Facebook or Reddit. I think they must count ~100 engineers?

I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!

10 comments
It's because they're just serving a big site, not running the world's most sophisticated surveillance and ad serving machine. Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought through architecture.
> Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought through architecture.

No.

1. Your comment makes it sound like Wikipedia is just, or mostly, serving read-only content, which is far from true. Yes, static read-only content is significantly easier to serve than dynamic, editable one, but Wikipedia is the latter.

2. Claiming it's easy to build something at this scale is "isn't all that hard" just makes me think you've never done anything similar. It reminds me of devs saying they could re-build MS Office over a weekend. It's just ignorant of the software's actual complexity.

I'm not associated with Wikimedia in any way, but have worked on large-scale software projects before, and things are quite different from, say, websites only serving 100k monthly active users.

I have, actually, worked on very large and interactive websites at the very core. Notably: betfair.com which has a very busy API and website and used to be something like a 1:10 write:read ratio with multiple clusters and layers of fancy caching to keep it all coherent down to millisecond scales.

Wikipedia does not need to be globally consistent like Betfair does and the ratio of writes to reads is nothing like 10%, I'd guess at one write per million reads or less. There are several pretty obvious ways to architect a site like Wikipedia for effectively unlimited scalability. The main trick is that it doesn't matter if a page is slightly stale and you can queue edits in the backend for quite some time (many seconds) without severely harming end users. Given those constraints it really isn't rocket science given the plethora of amazing tools we have to hand.

What I'm NOT saying is that I could build it in a weekend. It would clearly require a few teams of skilled engineers to put it all together and, crucially, operate it. My initial comment was in the context of Wikipedia having 100 engineers, and I think it's reasonable to say that a team that size is easily capable of such a feat.

Hi, I am interested in this project. Could you please provide some minor detail about the architecture, like what framework was used for serving that many requests?
I've never heard anyone in my life say they could rebuild MS office in a weekend.

What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?

I've never heard anyone in my life say they could rebuild MS office in a weekend.

The saying usually uses Facebook or Twitter.

That seems indeed comparable given the scope and functionality of Wikimedia products.
The core parts of Office could be done on a weekend, but in order to get the same complexity and incompatibility it would take several "codemonkeys" several years to achieve.
Your countering a point they didn't make. SQL to HTML templates indicated dynamic site, not static. From there, they describe surveillance and ad networks that both increase the browser workload and make it rely on 3rd-party dependencies.

I thought it was a good, but snarky, point. Especially given my browsing sped up after I installed extensions that turn all that crap off.

Please be careful of logical tautologies:

"It all scales in all directions with a properly thought through architecture" sounds dangerously like, "Programming isn't that hard if you just do it right."

> Programming isn't that hard if you just do it right.

That's not a tautology. In fact, it's actually worth pointing out, especially to junior engineers who get frustrated by how hard everything is, that it actually doesn't need to be that hard if you, well, do it right. Obviously that's not productive feedback without actually helping them be better, but it's far from a tautology.

For anyone wondering, a tautology is a statement that is logically true by construction, rather than contingently true because of the way the world is. For example, "Programming isn't that hard if it's easy" would be a tautology. Constructing a counterexample by changing programming to something else shows that this was not a tautology to begin with: "Sending a man to the moon isn't that hard if you just do it right," which is obviously false, because even if you do it right that's objectively difficult.

Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.

I think that you are speaking of a logical tautology, while I am speaking of a linguistic tautology. If I am correct about this, we are both right.

A logical tautology is, "A statement that is true by necessity or by virtue of its logical form."

A linguistic tautology is, "A phrase or expression in which the same thing is said twice in different words."

In formal debating, for example, you can call someone out for either type of tautology.

> That's not a tautology. In fact, it's actually worth pointing out, especially to junior engineers who get frustrated by how hard everything is, that it actually doesn't need to be that hard if you, well, do it right

But this boils down to If you build systems using a high level of skill and foresight, it's easy to do.

This is of course not a tautology, but a contradiction. I agree that inexperienced developers can, as it were, 'make life hard for themselves', but that's (trivially) due to their inexperience. I don't think there's a silver bullet for inexperience.

Over-engineering is bad, as is under-engineering. Fuzzy principles like 'YAGNI' can't be applied without skilled discernment, which means experience.

> Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.

I think I agree with this, but it depends on specifics. What sorts of things are you thinking of?

I appreciate what you're saying, but I don't think it quite applied. What I meant was that it's easy to create an architecture for an application that doesn't scale well at all. Eg - poorly sharded data, lots of cross dependencies etc. However, if you properly think through your data model and data flows and use cases, it's generally possible to create a system that is extremely scalable in all directions. This is certainly not easy, but it's a hell of a lot easier than creating some huge ai driven data slurping ad empire.
I totally agree that you make an excellent point about the relative ease/difficulty of various approaches.
>>"Programming isn't that hard if you just do it right."

Is this like saying, programming isn't hard if you choose easy enough problems to solve? Or should we ask for a link to see a demo of an AGI implementation?

I guess math is not hard either if you're "doing is right", as long as it's all arithmetic...

>>That's not a tautology.

I would agree tautology is not the best description, probably fallacy would do fine.

> Is this like saying, programming isn't hard if you choose easy enough problems to solve?

No, this is saying that things don't have to be as hard as we make them. You don't need more than a hundred people to run a top-ten website, and that shouldn't be surprising. It is surprising only because we are so good at making things overcomplicated.

But also perhaps it’s because they didn’t allow a team to endlessly iterate on tech minutiae until they required many teams to keep it running and iterate on tech minutiae.
> I think they must count ~100 engineers?

https://wikimediafoundation.org/role/staff-contractors/ has the names of 379 employees. I believe (perhaps astonishingly) that is all - engineers and non-engineers combined. Their engineers spread across departments, but judging by the 141 instances of the string 'engineer' in that page, I'd be surprised if the number exceeds 200.

Speaking as someone listed on that page, and having attended all-hands meetings, yeah, we're not huge.

Though it is worth bearing in mind that everything's open source, and there's a hefty community component. So there's a more vaguely specified number of people who might provide patches, and individual wikis are mainly run by volunteers.

Just drop by and say thanks you. Not many worldwide charity for human knowledge. Add oil.
That’s what happens I guess when you’re running a charity, you can recruit top talent (I assume many 10x folks wouldn’t mind working for wikimedia!) and every dollar counts. Pretty incredible.
They seem to be at the leading edge of hiring remotely and they don't pay anywhere near facebook salaries. The culture must be attracting some strong developers.
And they're hiring! https://wikimediafoundation.org/about/jobs/#section-8

I worked there for four years and I miss it every day.

> I worked there for four years and I miss it every day.

Sorry but now I'm curious, why did you leave?

Wikipedia has a huge impact in people's lives, particularly in non-English languages, and there's so much work to do, and so much of it feels urgent and necessary. I really responded to that, and I wasn't careful, and burnt myself out. (This was not the fault of the org; Wikimedia is largely a do-ocracy, and if you're intent on working through the small hours of the night, there is very little anyone can do to stop you. Co-workers who saw what I was doing did urge me to pace myself and exercise self-care.) By the time I realized what I was doing, I was in a pretty bad way, and felt like I needed a complete change of scenery to get back on my feet.
Not to diminish Wikipedia engineers talent, of course...

But, I'd consider Wikipedia traffic to skew heavily towards anonymous read-only, with very few logged-in write traffic.

This allows for tons of caching opportunities: Varnish, Memcache, etc. And these techniques are well known.

The proportion of read/write may skew towards reads, but Wikipedia still is an application where any user can create state visible to all other users. It's not as simple as this comment makes it out to be.
But how quickly must those writes be reflected in the reads of others? If you can accept a few minutes of latency there, I imagine things would get easier
In order for wikipedia's anyone can edit to work, its really important that when someone makes a bad edit to a popular article that it can be removed immediately. This is important both to get things fixed quickly and to make it less of a juicy target so less people vandalize (no fun to vandalize if it doesnt stay up).

I suspect latency in the minutes for cache updates would be unaceptable to wikipedia users

First off, keep in mind that even scaling a broadcast publication can be complex. Sure one can bolt on fastly or s3 but cache invalidation is never a simple problem.

Next "power users" as others put it are not a single set of editors. It's more of a social network with multiple levels of trust. The idea of a wiki is that all users have write access, even if those changes are moderated to have different levels of latency.

Of course there are ways to engineer the system, but at that point one is, well, engineering a system. And WMF is doing so on a shoestring compared to other comparable levels of traffic.

Is WMF creating new paradigms of computing? Probably not. But they are doing a good job, IMHO.

It must be immediate, because Wikimedia can detect edit conflict (when someone update the article you are in the middle of editing)
If you consider the amount of money they are burning in comparison to 5 years ago, are the results really that impressing? See https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...
I found this updated version: https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C...

Their expenses have doubled in less than 5 years...

Even if their ratio Expenses / Assets has now decreased compared to 3-5 years ago (but stalls now), it means that their goal of financial independence is still very far away and they still rely heavily on a huge amount of donations.

I buy that yes it takes time to be financially independent of donations while offering a global information service for free.

But that essay is clearly pure hyperbole. The expenses aren’t exponential, they’ve been roughly linear for a decade. Notice how the word exponential was removed in the second version. The graph is showing increasing savings along with increasing growth, and the expenses appear to have slowed slightly in the last five years compared to the five prior years. It’s completely failing to demonstrate the stated claim of runaway spending, the numbers practically prove the opposite.

Plus it’s not outlining what the money is used for, so there’s no concept of efficiency here, no reason to doubt that increased service came with increased expenses. There’s zero meat in this argument.

Whatever; last year’s total expenses seems very small to me compared to web sites of similar size; there are startups smaller than Wikipedia’s team that have raised more money than Wikipedia’s yearly expenses without managing to deliver anything. Wikipedia’s value to the world is currently larger than it’s expenses, IMO, and I think it’s impressive what this non-profit has done.

It’s sorta interesting how the replies to this digressed into linguistics about systems architecture, and nobody called out the “elite engineers” statement.

I’ve worked with a couple of engineers who are now on Wikipedia’s SRE team. They’re good engineers, but not elite by any means. Not “10x” developers or wizards in castles or whatever. Good solid engineers who I would work with again and fight to hire. But they’re not savants or even the top 10% of folks I’ve worked with. Solid mid to sr level engineers I’d be happy to hand a project off to with ambiguous goals and little oversight, and I’d expect them to get a team of 4 or so other engineers to be more productive.

These are the engineers who meet the job requirements for SRE positions.

I used to work at FB and now work at Reddit. The engineering staff count at Reddit is within the same order of magnitude as the number you cite above. :)
Yeah, but unlike on Reddit, I never see "something went wrong" on Wikipedia.

No offense to you nor your team, but to me, as a consumer, reddit's product doesn't appear nowhere near as polished as Wikimedia's projects.

No offense taken, I don’t work on the product side of things there.

Also, with the caveat that I don’t know enough about the implementation details of the product at Reddit: I’d argue that Reddit’s workload is more write heavy that Wikipedia’s workload, which makes caching and scaling a bit harder for Reddit, relatively speaking.

That could be true, but do you have some numbers? Wikimedia wikis are in the order of a few hundreds edits per minute, around a thousand and sometimes more. https://tools.wmflabs.org/wmcounter/ https://wikipulse.herokuapp.com/
Wasn't Instagram famous for having a very small team of engineers responsible for the availability of the entire platform before Facebook acquired them?
Not sure how big ig was on acquisition, but I always remember that story being about WhatsApp with only 35 engineers
Business Insider ran a story about them at the point of acquisition.

April 9, 2012 "Instagram was acquired by Facebook today for $1 billion in cash and stock. It only has 13 employees and a handful of investors. ... Meet 11 of the lucky employees and 9 investors behind Instagram. ... Two other employees were hired during South by Southwest last month and their information wasn't available for this story. "

https://www.businessinsider.com/instagram-employees-and-inve...

> I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!

Not only that. They do all this with amazing openness. Their records of incidents and deployments, who's in charge of what, rotation schedules are all public and shared in MediaWiki (although they're not that well organized). I can trace this back to circa 2005. Maybe this could be the largest knowledge base of devops that is public.

cf. https://wikitech.wikimedia.org/wiki/Category:Incident_docume... https://wikitech.wikimedia.org/wiki/Deployments/Archive/2019...

I was motivated to donate a small amount of BCH to WMF after reading this announcement.