Twitter, from day one, has been poorly architected. Defending that is pretty stupid and then to do it publicly is, well, kind of asking for it.
back
3 comments
I'm coming at this from a different angle. It should be possible to have a heated disagreement, even in public, and not have it result in a same-day firing. In fact, some countries have it explicitly on the law books.
All this does is create an environment where people are afraid to voice opinions and address issues, e.g. because they're afraid of shoot-the-messenger. In my line of work (I make cars), dysfunctional discourse environments like that can get people killed.
Good thing Musk isn't in charge of a car company
Sure, and I'm a big proponent of labor. Musk is certainly not.
That said, Twitter's architecture is a Rube Goldberg contraption and Musk is clearly on a warpath to fix it, so to come out publicly and yell, "It's fine," is kind of crazy. It's not fine and you're saying you're not interested in fixing it.
Looking at the exchange, he did confirm that it's slow on Android/iOS. He claimed it's for other reasons and gave examples. He might be wrong (I haven't looked at Twitter's architecture, commentary on the technical content of the exchange would be interesting), but I wouldn't say he said "It's fine".
He said it's "net spend" and that's another way of saying, "It's fine." It's the equivalent of admitting the engine is running hot, but saying, it's not a problem with the engine, it just needs lots of oil." Frankly, I think the developer is embarrassed. There's no reason for it to be as slow as it is, other than he and the other engineers simply don't know what they're doing.
I think Musk is calling it out because he wants to see who wants to fix it and who wants to defend it.
I think you misread "Ad spend" as "net spend" in his tweet (I can't locate any using the phrase "net spend"). He mentioned that performance work correlates well with "UAM and Ad spend", which to me reads as "it's worth working more on this problem" and is indicating the willingness you're looking for.
From what I can see, he offered an opinion that the number of client requests (he gives a number of 20) and the number of internal RPCs (he gives 200) is not the primary reason the app is slow. I can't tell you whether he's right or wrong.
Thanks for the correction. You're right, and I can't tell you either, so I shouldn't really argue.
FWIW, if he actually had written "just throw more money at it to make it go faster", I would understand your more negative opinion. :-)
Clearly these engineers have never done customer feedback for developing nations, twitter interactions clearly provide evidence for what musk said, but some engineers seem to be in denial, imho the firing is cause worthy.
Can you point to the "clear evidence" that "thousands of RPC calls" are the issue? Note that the dev didn't claim performance is good.
Performance has been bad forever. Twitter has had forever to fix it. They haven't. By definition, the existing developers aren't smart enough to do it. The developer here is trying to throw smoke because he's embarrassed. He should be.
Or it could be that previous management was constantly prioritizing new features instead of performance gains and now he's getting unnecessary flak for it.
and indeed current management has been prioritizing new features :P
The worst thing is that by firing and laying off too many people on a team with so much legacy code, they might end up scrapping the app and doing a rewrite because it'll be easier than fixing it. So inevitably, it will run faster, and people will hold it up as an example of Musk being a "genius" . When really, it's exactly what the guy suggested doing, but instead of listening they fire him and force a situation where they have to do it anyways.
Blind assumptions count for nothing. If you actually read the tweet thread Eric (the former employee) gives a summary of the trade offs made that have given Twitter bad performance and suggested removing specific features to improve perf.
So, "no".
It doesn't matter if it's "thousands of RPC calls" or not. Eric's response is, "It's definitely not that! It's... well... we don't know... so just spend more money on servers." That response alone would have me fire him.
> By definition, the existing developers aren't smart enough to do it.
You have no evidence to support this. Performance sucks in other countries because product leadership with their hands on the wheel of the company never prioritizes "unsexy" investments in performance and infrastructure work (same as the new guy btw). That is exactly what Eric was advocating in his thread.
I think that's misreading the exchange. He was challenging Musk on the fact that "1000 RPC calls" isn't what's happening and isn't the reason for the slowdown in his opinion, but did not deny that there are performance issues - and even cited other reasons it's slow. I don't see the denial you see.
Said engineer, said 1000 RPC calls are not the issue, and when questioned further on proving a tracepath / perf result casually exited the conversation, to me it looks like they are not ready to do better or listen to valid criticism, leading conversations on semantics is just creating noise instead of acting on signals.
Also 1000 RPC's are a very valid performance bottleneck, 1000 * latency caused by serde+ network can compounds easily based on data locality + container network orchestration, no data was provided otherwise even when asked for.
Except that you can open the dev console in a browser and load Twitter.com yourself and you won't find 1000's of XHR requests.
Not to defend any of the catastrophe of management that just happened but can't both be true? Eric does say in his tweets that a lot of time is spent waiting on the network. If that is because server side, the endpoint handling a request is spamming other backend services with 100s of RPC calls it would be slow. Now, why you would call out your android dev. on why your backend is not responding fast, I dunno.
That's client-side requests. Musk was talking about internal RPC calls to microservices needed to ultimately respond to the client-side requests.
If the internal arch is heavy on microservices and you're putting together, say, 50 tweets you're at 1000 RPC calls with just 20 per tweet. It's not inconceivable.
The developer puts the real number at 200.
RPC generally means server side calls, probably this https://twitter.github.io/finagle/, and XHR is not RPC.
Many software are like that, especially those that have been around for some time. You don't see the CEO of Microsoft spewing nonsense about Windows publicly, and then punching down on an employee for correcting him. There is no need for the circus. Technical debt is a fact of business, it can be dealt with professionally. It's not a reason to treat people badly.
The developer, Eric, is in the wrong here.
The product is especially bad. The new CEO is trying to reform it. Eric is saying "spend more," which is the refuge of bad architecture. That's not the answer. And instead of privately working together to review it, he's publicly arguing. I wouldn't accept that either.
He called Musk's public statement out, because it was wrong. Twitter shouldn't be the place for this, but it's on Musk for creating such unprofessional atmosphere in the first place.
Imagine the same thing with Microsoft. We all know Windows is held together with scotch tape, not to mention that 7 years after Windows 10 they still haven't quite manage to unify the old and the new interfaces, not that Windows 11's new interfaces are any good. What a mess. Except it's not, it's business as usual. Big software projects are messy and hard to manage, but these issues are unimportant when you get the bigger picture right. For Windows the bigger picture is that it runs all your application. This is what makes Windows a good product, the rest is details and can be fixed over time. Preferably in a professional manner.
How is Musk trying to 'reform' it when since he's owned it Twitter has had worse performance than ever, degraded features due to his bull-headed decisions and is literally burning money because he's chasing advertisers?
And the only one that started a public argument was Musk by making an incorrect claim and defacto blaming the engineers for something he clearly doesn't understand. He's trying to pull another Paypal and it's blowing up in his face now like it did then.
Eric didn’t defend the entire architecture though. Space Karen made a specific claim (“>1000 RPCs to render the time line”) and Eric refuted just that one claim.
Eric essentially said, "That's not why the engine is running hot, it just needs lots of oil." That's a stupid response.
No he didn’t. Eric essentially said “the clients don’t make >1000 RPCs to build the time line”, a direct refutation of Space Karen’s original assertion.
Yes, but he went on to say that it is truly slow and that it needs more spend, which is another way of saying, "You're wrong! It's not that! ... but I have no idea what it is," which isn't a very good response.
That’s a pretty uncharitable interpretation. I read it as “You’re wrong” (which is correct - Space Karen was wrong in his claim about why things were slow) “and we should figure out why, which will take time & effort (i.e. investment).”
You could be right. But how do you know the consultants Musk hired are wrong? How does Eric know? Maybe there was a situation found where that did, in fact, happen. Certainly with a large number of microservices, it could happen. Maybe Musk is saying "edge case" and Eric is saying "on average." Personally I feel outside consultants could have very well found problems that the internal team did not see.
What everyone agrees on is that Twitter's architecture is deeply broken. Eric's been there six years, so he's probably at least a little culpable in that. If I'm Eric, and was part of a team that fucked up that badly for six years, I certainly wouldn't argue back with that much confidence.
It’s not that hard to aggregate multiple APIs to a single request.
I guess it’s not that easy. Space Karen did just tweet the first thing he heard from his stuff. And it’s more clearly he doesn’t know how to do PR by his own.
It’s different to apologise for bad performance or blame implicitly the developers.
You’re assuming an awful lot.