Scaling

It’s easy to get sucked into partisanship, and on Bluesky I am long past being a neutral commentator. I tried and I failed. To my shame, emotionally I am team ActivityPub all the way, and I want AT to fail. Their success is my regret at this point. So my heart sank when I read this unambiguously good news in the latest essay from Cory Doctorow:

The good news here is that Bluesky has made enormous progress in true federation. The cost of operating a full Bluesky stack has fallen from tens of millions of dollars per year to tens of dollars per month…

If you can run Bluesky on a Raspberry Pi then there’s really no compelling argument to run an ActivityPub server any more. Of course you’ll need one of the fancy new Pis for AT, whereas AP runs cheerfully on a Zero. But this is a rounding error.

Breakthrough

From what I know of AT, it is truly surprising that they could have made such a breakthrough. I was aware that there had been a change recently which means relays are no longer archives, meaning they no longer needed vast storage. But that still leaves relays processing everything from the whole network. That sounds big, doesn’t it?

So I followed the link to see the technical details. Side note, I can’t actually see the article. There’s a GDPR-violating popup refusing to show me the article unless I agree to terms of use in Japanese. So I resort to the usual hackery to get the text. Once I get through that, I can see the important part:

What kind of a server do we need? As of April 2025, the atproto network firehose throughput hits a peak of around 600 events/second on a random day. In the past we have seen sustained rates of 2000 events/sec, and it is nice to have some headroom.

On our (large) production servers, the relay has been running steady-state with a couple CPU cores utilized, and up to 12 GByte or so of RAM. Memory usage starts low and creeps up; it is mostly consumed by identity caching, which can be configured down.

We see about 30 Mbps of sustained bandwidth for a firehose WebSocket. On the ingest side, that is the sum of all the PDS connections; on the output side, it is a single unified socket (one per consumer). Some overhead is good, both for traffic and recovering from downtime, so i’d recommend looking for a server with 200 Mbps unmetered.

Yup. You can run your own relay, but it still needs to process every single event from the entire network, immediately when it happens. It needs to have continual incoming connections from every single PDS out there. And yes, this does fit on a Raspberry Pi, in April 2025, with about 6 million monthly active users. But when the network hits a billion users you’ll need a hundred times that.

Growth

This is setting people up for tragedy. Suppose I’m in a group of 100 people and I propose setting up a Bluesky stack for them all to use. It doesn’t cost much, so perhaps I pass around a hat, or I offer to just eat the cost myself. Seems like a friend of Cory Doctorow has done exactly this just for him. It works great, and 100 people use it quite happily. But in 2026 the costs double, even though there’s still only 100 people using it. In 2027 costs double again. And again. The number of users of my server, the number of posts they make, the number of connections they have, those all stay small and stable. But I have to pay for the growth of the rest of the network, even if it’s irrelevant to me. I’m gradually being suffocated by other users’ decisions, trapped by a promise I made to my friends. The best case is that early on I admit that I was wrong, shut down the service, and help them find a commercial alternative – a long-term financial drain that I forced them into. The worst case is that I destroy myself trying to keep the service alive.

Now, as it happens Bluesky growth seems to be stalling already. So perhaps running your own stack will be fine in the long term. But I also believe that 6 million people isn’t enough. In particular, it’s not enough for my use case, which is maintaining contact with a network of friends located around the globe.

Of course we can’t boast too much in the Fediverse, which only has around 1 million monthly active users, and hasn’t grown for years. And many people in the Fediverse actively want to prevent it getting much bigger than that. But I believe the Fediverse does have plenty of room to grow. A billion users is achievable. And that dream is not incompatible with those who like the current sense of coziness – the system can accomodate any number of backwaters and side-channels. So how does scaling in the Fediverse improve on AT?

Not actually very well, it turns out.

Stampede

I’ve been thinking recently about the Mastodon Stampede problem. In particular, Jamie Zawinsky is an important test case in showing up the limitations of the WordPress ActivityPub Plugin. He wants pretty much exactly what the plugin provides: his blog entries pushed into the Fediverse, and comments pulled back and displayed on his site. But he can’t use the plugin and instead wrote his own solution. WordPress is fairly slow, and his blog is popular enough that he can’t afford to allow Mastodon to automatically hit his server in a coordinated way. Caching could help him, but he chooses not to enable this because (in my limited understanding) it leads to pages being stale. Anyway, caching is a band-aid, temporarily hiding the brokenness of the underlying architecture.

Instead, his approach is to block Mastodon from his site entirely. He has a separate process that pushes his blog entries to the Fediverse, and copies comments back onto the blog, asynchronously. That’s one way to do it.

Pushing and Pulling

So I thought hard about how the WordPress plugin copes with this problem, and what more it needs to do. My conclusion is that I think it actually does improve on the baseline “I have a WordPress blog” situation, because it pushes articles instead of waiting for Mastodon to pull them. But then if someone else boosts your blog post, the same problem shows up. Is this good enough? No.

Suppose you have 10k followers, spread across 2k servers. You post a link to your latest blog entry. That tweet is very small, and the push happens in a tight loop on your server using the same payload, so it shouldn’t need much resources. On the other end 2k servers might need to spin up a connection and a handler, but that’s their problem. This might take what, ten seconds? But then, each Mastodon server will assume that their users will want to see the article. So each of them responds with a query for the entire article, including any images. Now, it seems images on WordPress are served pretty quickly, but assembling an article is very expensive. So 2k requests within ten seconds sends the server straight into thrashing overload, and for a full minute that site is unusable.

With the WordPress plugin, this goes differently. When you publish the blog entry, that triggers the server to push the entire article into the Fediverse. That again happens in a tight loop. The data is much bigger, so perhaps this does require more resources. But that load should be squeezed through just one thread, leaving most of the server free to respond to other requests. The push does not include images, so the receiving servers will probably immediately request those. But images are quick to serve. Assuming the receiving servers are implemented reasonably, that they actually use the received data instead of requesting it again for no reason, this should be fairly quick, and not overload the sending server.

However. Suppose that on one of those servers, a different user with 10k followers boosts the article. In Mastodon, boosts do not include the full text of the article, only a link. Now, those 10k followers are also distributed across 2k servers. But it’s a different 2k. Probably 1k of those already have the article, and again assuming they’re implemented reasonably, they will not hit the original server again. But that still leaves 1k requests, enough to send the original server into thrashing overload.

Forgery

A sensible solution to this is for boosts to include the full article. The spec certainly allows that: when I boost Jamie Zawinsky’s post, I use an Announce activity, with myself as actor, while the object is his Note, and has its own separate actor.

The only problem here is that the receiving server needs to trust that I accurately reproduced his Note. I could have forged it. I don’t see that as being a big problem. People happily exchange screenshots from other social networks, and forgeries would be quickly spotted. That’s especially true when by definition these messages come with a URL you can use to check the accuracy. A bit worse though is that not every ActivityPub server supports the same fields. There may be extensions that my server doesn’t understand, which it automatically drops.

All ActivityPub messages are signed to prove authenticity. My boost message will be signed by myself. It should be possible for boosts to include a second signature for the included object, copied from the original message I’m boosting. The snag is that practically all ActivityPub servers throw away the “envelope”, including that signature. It would be a huge change to the behaviour of servers to support reproducing upstream signatures, since they would have to keep a copy of the original plaintext message.

Still. I don’t see why we couldn’t start including the boosted text anyway, regardless of the small problems I described. Perhaps some paranoid or completist receiving servers would want to gradually fetch original copies too, just to double-check. But this can be a delayed process, so that the originating server doesn’t get overwhelmed.

Small Scale

All of that is a problem for big accounts. Compare that to my own experience. I run Friendica on a tiny VPS. I have just 65 followers, so publishing is a trivial load. If I ever wrote something that went viral instantly, my VPS would fall over and probably not come back up for weeks. And I have written tweets that went slightly viral, getting maybe 50 boosts. But each of the boosters only has a few hundred followers. In each case, a majority of the accounts they boost to already have my tweet. So as long as the boosts come a minute apart, my tiny VPS can cope with this. I have also occasionally had a tweet boosted by a minor celebrity account with 10k followers, and that too seems to work fine. It’s possible that in fact this kind of thing does overload my server, but since I only use my Fediverse account asynchronously, I wouldn’t even notice.

Unlike with AT protocol, I never need to worry about the growth of the network leading to exponentially increasing load and costs. I do hope that as the network grows more of my friends will be included. But a doubling of the network size won’t double the number of my contacts. It’s more like a logarithmic relationship between the network size and the number of contacts I have, and traffic then scales in proportion to that number of contacts. Ultimately I’m just one person, there’s an upper limit on how many real friends I can have. And each of those is a real person, with an upper limit on how much they can post on social media.

However, while the problem is reduced in the Fediverse quite dramatically, it’s not two or three orders of magnitude. Even a small server still needs to communicate with a pretty large portion of the network in a small space of time. A billion-person Fediverse would still regularly destroy small servers when a tweet goes viral. And very popular accounts still predictably destroy their own servers. In the early days of the Mastodon migration Paul Krugman attempted to use the Fediverse, and it was a catastrophe. He won’t be back soon.

Choose Your Corporation

Bluesky works today because in practice there’s only one important relay. If it became as decentralised as the Fediverse, the advantage would evaporate. Every additional relay improves the decentralisation of the network, but imposes a huge load on every PDS in the system. Every additional PDS improves the decentralisation of the network, but imposes a huge load on every relay.

This isn’t necessarily a problem. You don’t really need tens of thousands of servers to achieve the benefits of decentralisation. The difference between two choices and one choice is huge. The advantage beyond two is kinda meh. And I guess that’s the BlueSky vision. It’s about bringing choice to the masses, not bringing self-hosting to the masses. Just like today you are free to choose between fealty to Google or Apple, in the future you will be able to choose between Bluesky and Blacksky, or one of a small menu of other options. Compared to the wider economy, dominated by monopolies, a choice between two or five or twenty providers seems like a pretty competitive landscape.

Censorship

But as a social network, we need to think about censorship, and here is where conventional ideas of free market competition break down. I’ve said before, labellers are a distraction, the important censorship happens in the relay. If there’s only a small handful of relays, then there’s very little room for diversity in policies. They all must be considered “general purpose”.

We’ve seen this play out with Mastodon: there is a striking convergence in the terms of service among all the >100k servers. They are big because they aim to be big, which means a correspondingly light touch when it comes to censorship. But they will all instantly bow the knee to government control, because they can’t allow vast numbers of users to be punished for the sins of a handful of renegades. And then if you’re so big that you’re systematically important like this, you cannot kick off groups you don’t want without being accused of bias. Balancing all of this is very hard, and doesn’t leave you much space to choose a position. That’s why the really valuable communities on Mastodon are smaller, allowing for a diversity of carefully-tailored codes of conduct.

The Bluesky approach seems likely to lead to a kind of “strip mall diversity”, providing the freedom to choose however you want between KFC and Burger King.

And to reinforce this again, censorship happens in the relays, not the labellers. Yes, the labellers will allow you to impose your own preferences on your feed. But they are only coloured glasses. Everything seems rosy to you, but that’s only to you as an individual. The relays can and must impose a more fundamental set of policies on the network. As the network grows, it’s not just possible but likely that we’ll see a repeat of the female-presenting nipples controversy. But at the same time Bluesky requires me to use infrastructure that fuels and profits from anti-LGBT hate speech. I shouldn’t have to accept either of those things.

Internet Of Things

One of the more baffling blind spots about Bluesky is their choice to be a microblogging platform. Baffling to me, that is. I guess people who actually used Twitter would understand better.

But microblogging, i.e. Twitter and people not cool enough to be on Twitter but who would like to be, is a niche pursuit. At its peak Twitter had a couple hundred million users. Statistically indistinguishable from zero. You’re basically never gonna make friends with anyone who’s ever used Twitter. That’s the pool of users that BlueSky is targeting.

Real people use Facebook and, to a slightly lesser extent, Instagram. Neither of those is a microblogging platform. But both of those are targets of ActivityPub. And more. ActivityPub also targets news, explicitly, with the Article object type – so already a far bigger audience than microblogging ever had.

But it has ambitions so much greater than that. ActivityPub is extensible. It uses JSON-LD because this allows remixing with any kind of communication anyone could come up with. This facility is tragically underutilised today, but just take the classic example of a chess game. A system that scales to 600 events/second might be fine when everyone’s microblogging. But include every move in every public chess game in the world. Then every public backgammon game. Then every public Warhammer 40k game.

Suppose we implement a wiki on ActivityPub. This is quite feasible, you just allow Update activities on Articles by people who aren’t the original author. But then, imagine extending this to something more like Wikidata. Instead of textual articles, we assemble detailed technical specifications about, well, just about anything. Every time someone documents the fact that some Pokemon is a subtype of some other Pokemon, this is another message in the network. Now we’re talking serious event rates.

We already have ForgeFed, which promises to put every bug report and pull request onto the Fediverse. Like most people, I spend most of my time working, not posting on the Internet. “Working” for me largely involves fidgeting with bug tickets. You’re going to get a much higher event rate from my work than from my occasional textual musings to the world. And if we replicate this across all the other ways people work, it’s going to overwhelm the events from microblogging.

Even this is fine, if you only have one relay. But if the number of relays scales with the number of users… well, it just doesn’t.

This is a reality that does not exist today. But I really believe, and hope, that it will exist tomorrow.

Bluesky also targets extensibility, in that it has lexicons. It’s still unclear to me how exactly the extensibility is supposed to work. Do we expect PDSs to accept messages with all lexicons? How do they implement graceful degradation for lexicons they don’t support? But clearly the core idea is the same. Like ActivityPub, it seems no-one is really taking advantage of this yet. But it seems that the idea is deliberately much more constrained than the ActivityPub equivalent. Given their chosen example of a recipe lexicon, it’s likely that this will only be used for relatively lightweight applications.

Just as Bluesky is focused on just one type of social networking, their vision for extensibility also seems very limited. Those assumptions get into the architectural design too, constraining what Bluesky can ever achieve.

What’s In A Protocol?

One thing that bothers me though: what is it about AT that makes it different to AP?

There are relays in ActivityPub. They’re not so commonly used, but they exist. They work over the same protocol just like anything else.

So how come AT can’t work the same way as ActivityPub? PDS’s have to connect to multiple relays, right? Why can’t one PDS act as the relay for another PDS? That way PDSs could exchange direct endpoints as part of adding a contact. There would be no need for a relay at all, unless the user chooses it. And this would instantly make Bluesky look like ActivityPub, with all the same limitations and potential.

I haven’t looked into the details of how AT works, perhaps this is answered somewhere. But even so, specs can be altered. AT is open source, it’s not like the Bluesky company owns it. Right?

Stalemate

Thinking about how this plays out, it seems unlikely that my partisan emotions will ever be satisfied with a clear victory over (or defeat by) Bluesky. If Diaspora is still around, Bluesky won’t be going away soon.

It seems likely that Bluesky will stabilise as a popular microblogging platform. As such, it will top out at a few hundred million users. That will make running a relay pretty significantly expensive, outside the range of a hobbyist or local community. I expect a small handful of relays to survive, and funding those will be a continual challenge. But the costs will be moderate compared to the number of users, so one way or another it won’t ever go under.

Meanwhile, I believe ActivityPub will primarily occupy the territory that Bluesky leaves behind. There’s already plenty of that: software specialised for photos, videos, music, forums, as well as long-form content like this blog. Ultimately there are far more real people in that territory, and their work is far more impactful. In microblogging, Mastodon isn’t going away either, and bridges will make the boundary between the two networks irrelevant. Therefore, Bluesky is in fact no obstacle in the path of ActivityPub’s growth.

So if you can run a relay on a Raspberry Pi in 2025, good for you. That is unquestionably good news.

Leave a Reply

Your email address will not be published. Required fields are marked *