Federation issues with lemmy.ml
Federation issues with lemmy.ml
We're aware of ongoing federation issues for activities being sent to us by lemmy.ml.
We're currently working on the issue, but we don't have an ETA right now.
Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don't seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.
Right now we're about 1.25 days behind lemmy.ml.
You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.
We'll update this post when there is any new information available.
Update 2024-11-19 17:19 UTC:
Federation is resumed and we're down to less than 5 hours lag, the remainder should be caught up soon.
The root cause is still not identified unfortunately.
Update 2024-11-23 00:24 UTC:
We've explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven't really made any progress other than ruling out various potential issues.
We're currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren't able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.
As of this update we're about 2.37 days behind lemmy.ml.
We are still not aware of similar issues on other instances.
You're viewing a single thread.
I mean.....if you wanted to defederare from lemmy.ml I'd be fine with that.
It's certainly an efficient way to resolve the problem.
There's too much content on lemmy.ml to defederate. We'd lose like a quarter of all the content.
We would adapt, just like when Beehaw was a significant amount of the userbase on Lemmy and they cut us and sh.itjust.works off, we adapted and they got smaller. Lemmy.world is a bigger server than lemmy.ml. Only reason their communities are still so big is network effect. Which would be curbed by them being cut off. As I've said already, network effect is curbed by force, taking away a choice, not providing 7 more choices while leaving the original.
We'd lose the .ml version of that content.
But I thought the whole point of federation was that if mods, or communities become problematic, you could always create your own. Then all the non-problematic people will move to that.
Is that not the very foundation of the concept of the fediverse?
It's not wrong, but I don't regard Lemmy.ml as being problematic enough to defederate from. Their moderation practices are questionable and their user base is annoying but it's otherwise generally tolerable. People can block the instance if they don't want to see content from it.
It would be nice to have a simple way to block the user base too
I wouldn't.