Why Twitter is doomed

So I have been watching the resignation notices for people leaving Twitter. Basically, it looks like all of the developers other than those on immigration visas are leaving. I see a lot of familiar names there, people who are experts in scalability who have published papers on how to scale things, experts in Internet protocols whose names are on IETF RFC’s, experts in algorithmic complexity with journal papers, and then just the average everyday Joes who keep the lights on. And I think: Twitter is going to crash.

Oh, not right now. See, here’s the thing. Everything at Twitter is massively redundant. There’s three or four copies of every tweet geographically distributed across multiple data centers. There’s dozens of circuits and routers into each data center. There’s two electrical circuits and two power distribution units into every rack. And so on and so forth. But:

Things break. Routers fry or lose their mind. Disk drives die. Switches quit switching. Hypervisors quit hypervising. The breaker blows on a PDU because it’s gotten old or hot and no longer will carry the load. Someone has to go in and fix these things as they happen. Sure, there’s redundancy. There’s other disks with copies of data. There’s other routers. There’s other circuits. There’s other servers with hypervisors on them. But as racks start going dark because nobody fixed things, the service slows down. Eventually the spare resources for redundancy run out. Then something goes out… and there’s nothing redundant to pick it up. Something crashes.

And if it’s an important part of Twitter like the globally distributed data store master index, Twitter might never come back up. 

I’m not sure how to describe this to non-technical people, but basically, consider it like the master index for all of Twitter (that’s not exactly how it works, but close enough). This is replicated in RAM caches worldwide. And if you crash it, it’s possible to rebuild it by scanning the actual key-value store for the tweets… but the people who know how to do that have gone out the door, along with the Joes and Janets who know how to fix the hardware. Maybe Mahesh and Yang here on H1B visas and thus unable to leave Twitter can reverse engineer it and over the course of a week or two figure out how to rebuild the indexes. But by that time the advertisers have turned off the money spigots, the users have fled to Reddit or Instagram, and Twitter has gone the way of MySpace.

At this point, I don’t know of any way that Twitter avoids that fate. Maybe Musk could go hat in hand and hire back some of the talent he just drove away. But I don’t think they’ll come back. Most worked at Twitter because they wanted to work at Twitter, not because they have to work at Twitter. Many of the people who know the most about the deep dark innards of Twitter made a fortune on the Twitter IPO and have no need to work anymore. And they’re likely to give Musk the finger.

The wildcard in all this is the bankers. Musk didn’t fund the purchase of Twitter with his own money. He borrowed money, or got other investors to come in on it. And those people are going to be *pissed* when Twitter crashes. Will they fire Musk and try to mend fences themselves with the people necessary to get Twitter back up and going? Or did Musk put enough poison pills in the contracts to keep that from happening.

Either way, Twitter is still dead. The contract stuff just determines who gets sued over it, and whether Twitter *stays* dead after it crashes that first time.

So it goes. I’m just gonna stock up on the popcorn and make bets about whether Twitter lasts longer than this lettuce:

— Badtux the Tech Nerd Penguin

Advertisement