I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!
No.
1. Your comment makes it sound like Wikipedia is just, or mostly, serving read-only content, which is far from true. Yes, static read-only content is significantly easier to serve than dynamic, editable one, but Wikipedia is the latter.
2. Claiming it's easy to build something at this scale is "isn't all that hard" just makes me think you've never done anything similar. It reminds me of devs saying they could re-build MS Office over a weekend. It's just ignorant of the software's actual complexity.
I'm not associated with Wikimedia in any way, but have worked on large-scale software projects before, and things are quite different from, say, websites only serving 100k monthly active users.
Wikipedia does not need to be globally consistent like Betfair does and the ratio of writes to reads is nothing like 10%, I'd guess at one write per million reads or less. There are several pretty obvious ways to architect a site like Wikipedia for effectively unlimited scalability. The main trick is that it doesn't matter if a page is slightly stale and you can queue edits in the backend for quite some time (many seconds) without severely harming end users. Given those constraints it really isn't rocket science given the plethora of amazing tools we have to hand.
What I'm NOT saying is that I could build it in a weekend. It would clearly require a few teams of skilled engineers to put it all together and, crucially, operate it. My initial comment was in the context of Wikipedia having 100 engineers, and I think it's reasonable to say that a team that size is easily capable of such a feat.
What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?
The saying usually uses Facebook or Twitter.
I thought it was a good, but snarky, point. Especially given my browsing sped up after I installed extensions that turn all that crap off.
"It all scales in all directions with a properly thought through architecture" sounds dangerously like, "Programming isn't that hard if you just do it right."
That's not a tautology. In fact, it's actually worth pointing out, especially to junior engineers who get frustrated by how hard everything is, that it actually doesn't need to be that hard if you, well, do it right. Obviously that's not productive feedback without actually helping them be better, but it's far from a tautology.
For anyone wondering, a tautology is a statement that is logically true by construction, rather than contingently true because of the way the world is. For example, "Programming isn't that hard if it's easy" would be a tautology. Constructing a counterexample by changing programming to something else shows that this was not a tautology to begin with: "Sending a man to the moon isn't that hard if you just do it right," which is obviously false, because even if you do it right that's objectively difficult.
Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.
A logical tautology is, "A statement that is true by necessity or by virtue of its logical form."
A linguistic tautology is, "A phrase or expression in which the same thing is said twice in different words."
In formal debating, for example, you can call someone out for either type of tautology.
But this boils down to If you build systems using a high level of skill and foresight, it's easy to do.
This is of course not a tautology, but a contradiction. I agree that inexperienced developers can, as it were, 'make life hard for themselves', but that's (trivially) due to their inexperience. I don't think there's a silver bullet for inexperience.
Over-engineering is bad, as is under-engineering. Fuzzy principles like 'YAGNI' can't be applied without skilled discernment, which means experience.
> Programming is hard, but we make it much harder than it has to be by doing it spectacularly wrong in many ways, both individually and collectively.
I think I agree with this, but it depends on specifics. What sorts of things are you thinking of?
Is this like saying, programming isn't hard if you choose easy enough problems to solve? Or should we ask for a link to see a demo of an AGI implementation?
I guess math is not hard either if you're "doing is right", as long as it's all arithmetic...
>>That's not a tautology.
I would agree tautology is not the best description, probably fallacy would do fine.
No, this is saying that things don't have to be as hard as we make them. You don't need more than a hundred people to run a top-ten website, and that shouldn't be surprising. It is surprising only because we are so good at making things overcomplicated.
https://wikimediafoundation.org/role/staff-contractors/ has the names of 379 employees. I believe (perhaps astonishingly) that is all - engineers and non-engineers combined. Their engineers spread across departments, but judging by the 141 instances of the string 'engineer' in that page, I'd be surprised if the number exceeds 200.
Though it is worth bearing in mind that everything's open source, and there's a hefty community component. So there's a more vaguely specified number of people who might provide patches, and individual wikis are mainly run by volunteers.
I worked there for four years and I miss it every day.
Sorry but now I'm curious, why did you leave?
But, I'd consider Wikipedia traffic to skew heavily towards anonymous read-only, with very few logged-in write traffic.
This allows for tons of caching opportunities: Varnish, Memcache, etc. And these techniques are well known.
I suspect latency in the minutes for cache updates would be unaceptable to wikipedia users
Next "power users" as others put it are not a single set of editors. It's more of a social network with multiple levels of trust. The idea of a wiki is that all users have write access, even if those changes are moderated to have different levels of latency.
Of course there are ways to engineer the system, but at that point one is, well, engineering a system. And WMF is doing so on a shoestring compared to other comparable levels of traffic.
Is WMF creating new paradigms of computing? Probably not. But they are doing a good job, IMHO.
Their expenses have doubled in less than 5 years...
Even if their ratio Expenses / Assets has now decreased compared to 3-5 years ago (but stalls now), it means that their goal of financial independence is still very far away and they still rely heavily on a huge amount of donations.
But that essay is clearly pure hyperbole. The expenses aren’t exponential, they’ve been roughly linear for a decade. Notice how the word exponential was removed in the second version. The graph is showing increasing savings along with increasing growth, and the expenses appear to have slowed slightly in the last five years compared to the five prior years. It’s completely failing to demonstrate the stated claim of runaway spending, the numbers practically prove the opposite.
Plus it’s not outlining what the money is used for, so there’s no concept of efficiency here, no reason to doubt that increased service came with increased expenses. There’s zero meat in this argument.
Whatever; last year’s total expenses seems very small to me compared to web sites of similar size; there are startups smaller than Wikipedia’s team that have raised more money than Wikipedia’s yearly expenses without managing to deliver anything. Wikipedia’s value to the world is currently larger than it’s expenses, IMO, and I think it’s impressive what this non-profit has done.
I’ve worked with a couple of engineers who are now on Wikipedia’s SRE team. They’re good engineers, but not elite by any means. Not “10x” developers or wizards in castles or whatever. Good solid engineers who I would work with again and fight to hire. But they’re not savants or even the top 10% of folks I’ve worked with. Solid mid to sr level engineers I’d be happy to hand a project off to with ambiguous goals and little oversight, and I’d expect them to get a team of 4 or so other engineers to be more productive.
These are the engineers who meet the job requirements for SRE positions.
No offense to you nor your team, but to me, as a consumer, reddit's product doesn't appear nowhere near as polished as Wikimedia's projects.
Also, with the caveat that I don’t know enough about the implementation details of the product at Reddit: I’d argue that Reddit’s workload is more write heavy that Wikipedia’s workload, which makes caching and scaling a bit harder for Reddit, relatively speaking.
April 9, 2012 "Instagram was acquired by Facebook today for $1 billion in cash and stock. It only has 13 employees and a handful of investors. ... Meet 11 of the lucky employees and 9 investors behind Instagram. ... Two other employees were hired during South by Southwest last month and their information wasn't available for this story. "
https://www.businessinsider.com/instagram-employees-and-inve...
Not only that. They do all this with amazing openness. Their records of incidents and deployments, who's in charge of what, rotation schedules are all public and shared in MediaWiki (although they're not that well organized). I can trace this back to circa 2005. Maybe this could be the largest knowledge base of devops that is public.
cf. https://wikitech.wikimedia.org/wiki/Category:Incident_docume... https://wikitech.wikimedia.org/wiki/Deployments/Archive/2019...