Lemmy.ml is blocking all requests from /kbin Instances
I discovered yesterday evening that Lemmy.ml is blocking all inbound ActivityPub requests from /kbin instances. Specifically, a 403 'access denied' is returned when the user agent contains "kbinBot" anywhere in the string. This has been causing a cascade of failures with federation for many server owners, flooding the message queue with transport errors.
This doesn't appear to be a mistake; it has been done very deliberately, only on Lemmy.ml. Lemmy.world and other large instances do not exhibit the same behavior. It also isn't a side effect of the bug introduced in Lemmy 0.18. You can observe by sending the following in a terminal
Additional evidence of this not being a Lemmy 0.18 bug:
This occurs when making web requests to any location on the Lemmy.ml webserver, not just ActivityPub endpoints.
Go to https://fedidb.org/software/lemmy and pick an instance running 0.18.0. Perform the above commands, replacing the URL for Lemmy.ml with that particular instance's address.
If this continues, my instance may need to defederate from Lemmy.ml. This is especially problematic because Lemmy.ml continues to federate information outbound to other kbin instances while refusing to allow inbound communication from them.
Spoofing the user agent is less than ideal, and doesn't respect Lemmy.ml's potential wish to not be contacted by /kbin instances. I don't post this to create division between communities, but I do hope that I can draw awareness to what's going on here. Defederating /kbin instances entirely would even be better than arbitrarily denying access one-way. This said, we should all attempt to maintain a good-faith interpretation until otherwise indicated by the Lemmy developers. It's possibel that this is a firewall misconfiguration or some other webserver-related bug.
It's possible that this is a consequence of the latest Lemmy update, in which a lot has changed. I have noted that kbin has some issues with request signature in communication with certain instances. I will try to check it tomorrow first thing in the morning.
Wow, thank you for the quick communication. Amazing aptitude. Almost nowhere else do you find a lead dev in the comment trenches letting us know what's happening, I'm kinda baffled.
I've noticed we are also not federating with lemmy.wtf for some reason, maybe it's something related. They are connected well with other lemmy instances but kbin can't get a hold of them despite being on their list of linked instances
If they don't want us to communicate with the users of their server, they should be defederated. Sucks for the users, but they can hop to another server whose admins don't break communication inelegantly and ghost admins who want to know what's going on.
If it's just a bug, then obviously we should wait for it to be fixed, but if they're not even saying they'll look into it, I don't have much hope.
At first I thought it was maybe a config error to block bots. But after some testing I see that the 403 is only given when the word "kbinbot" is in the user agent. String that just return the normal response I tested are "testbot", "kbinbo", "binbot".
Maybe it is some kind of automated rule 'block bots that cause lots of traffic'. kbin is becoming more and more significant, so it probably causes some non-insignificant load on Lemmy servers. Legitimate load, but the IDS rules would not know that.
I think the approach of changing the user agent in the interim is a good solution, if there's a technical reason they don't want kbin traffic they can be adults and reply in the chat, not just sneakily blocking incoming traffic
Doesnt surprise me, the developers of lemmy (which are owners of .ml) have an agenda. They are into censoring on ukraine news, and other stuff. I dont get why people are choosing lemmy over kbin when they are equally bad looking
Its a huge downgrade from reddit in terms of visual quality. But im soeaking on mobile safari so i guess that will explain alot but anyways heres what i see.
The posts that have pictures dont really show that well until you click on it. You need to scroll down a whole thread to comment on a post. Creating a post/thread is confusing, its hard to navigate “magazines” (idk why theyd call it that), theres a lot of problems with it.
Lemmy looked pretty bad on mobile safari, far worse than kbin
Eh? What are you talking about? There's literally a bunch of Russia and Ukraine news at https://lemmy.ml/c/worldnews. This rumor mill needs to stop. People started some bad rumors about Lemmy devs just because of their political association and now its blown up into full lies.
I use both Lemmy and Kbin, but I prefer Lemmy for a bunch of different reasons:
Interface is cleaner and more performant.
Getting to your list of subscribed magazines in kbin is a pain in the ass. Why is it buried in Settings?
Kbin votes are public. This is just bad for privacy. It's one thing to say that votes are public due to federation, and whole other thing to blatantly show it in the UI. There's a reason why voting at a government level is kept secret, just sayin'.
Also why the hell are they called magazines? None of the names in kbin map to existing conventions. "Threads" are posts. "Magazines" are communities or groups. I never get used to it because it isn't natural to call these things the way kbin calls them.
Posting this in light of a recent one week Reddit ban I earned for shitting on US police, as I'm sure many of us have gotten in recent weeks.
So I've spent the past few months working on a self hostable, federated, Reddit alternative called Lemmy, and it's pretty much ready to go. Unlike here we'd have ultimate control over all content, and would never have to self censor.
Obviously as communists, we agitate where the people are, so we should never abandon Reddit entirely, but it's been clear to all of us from day one, that communities like this stand on unsteady ground, and could be banned or quarantined at any moment by the white supremacist Reddit admins. This would be both a backup and a potentially better alternative. Moderation abilities are there, as well as a slur filter.
I wanted to ask ppl here if they'd like me to host an instance, and mod all the current mods here.
My concern is with this line: Obviously as communists, we agitate where the people are. I'm pretty left-leaning myself (I draw the line at authoritarianism though), but they're very open about using their platform to push an agenda.
That post was made by the founder of Lemmy and the instance they mention at the end became Lemmygrad (because lemmy.ml and Lemmygrad are the same people - the ".ml" in "lemmy.ml" even stands for "Marxist-Leninist").
With Lemmy.ml as the "main" instance, you're exposing all these posts to an admin team who has openly said "we are agitating for our views wherever the people go, and we have ultimate control over all content." They have already removed posts critical of China as being "orientalist" (which shows a severe lack of understanding of what "orientalism" even is - it's not "anything that criticizes our Dear Leader"). I can't trust that any community hosted on lemmy.ml is free of bias.
It isn't "rumors" when you can clearly go and back it up with a source.
(FWIW - I don't mind Lemmy as a platform. My Lemmy account is actually from 2020, before I realized what they truly were. The maintainers have done a good job of keeping politics out of the software, and the concept of only using "ethical" things is a myth when your computer has rare earth elements in it. As far as instances go, Lemmy.world seems fantastic and has grown to be far larger than the tankie instances. But Kbin is more fully-featured and has a lot more long-term potential, even though it's lacking in the short-term.)
If this is intentional, I have to wonder why they're doing it in such a troublesome way rather than defederating properly. If they want to defederate, so be it, but then just do that.
Well, lemmy.ml is the lemmy devs. Kbin is a competitor to their software. They might hate that Kbin is growing really fast and I think is overall better than straight lemmy.
I was almost ready to say that maybe they're just looking for "bot" in the UA. But, no. It must be "kbinbot" anywhere in the UA, and case insensitive. So, pretty deliberate it seems.
If this continues, my instance may need to defederate from Lemmy.ml.
That probably isn't the right fix for the long term, because otherwise it leaves kbin vulnerable to being DoSed by another instance intentionally returning 403s, or breaking down if something goes wrong with a lemmy or kbin instance that causes it to throw errors for totally innocent reasons. Kbin should be fixed to handle getting 403s reasonably-gracefully.
EDIT: Also, might be a good idea to check lemmy's handling of getting 403s as well for the same reason.
EDIT2: I don't know if lemmy/kbin have any kind of test code to test how well they deal with federated servers malfunctioning, but my guess is that using one set of tests for both would probably save effort.
I don't think it's assuming anything to say that they deliberately blocked user agents containing the text "kbinbot" (case insensitive). I think it's fairly clear that's exactly what has happened here. Instructions to test as many variations as you want yourself are right in the main post in the thread.
Now, what we cannot say is whether it was done with malicious intent. There's some plausible reasons they might do it that are not malicious.
Ideally it's a config error at the firewall. I saw an interesting idea posed on the lemmy post suggesting that it may have been targeted by a DDoS that used kbinbot in the user-agent string.
Ultimately, it's not happening at a code level, it's absolutely happening at a firewall level (nginx, which, for those who don't understand, is kind of acting like the door lock on your apartment building, where you need to go through the main security before you can get to your own place. Sort of the same idea here). I just spent a bunch of time testing a bunch of various user-agent strings, and it very specifically is matching "kbinbot". No wildcarding within the word, but it a string like "blahbinbotblah" will 403, whereas "blahkbbinbotblah" won't (and various other forms, like k.bin.bot, or kasdfbinasdfbot.)
It's pretty specific. Anyone who deals with firewalls in any capacity understands how nginx works, and specifically why they and others are raising an alarm.
So yeah, ideally it's a misconfiguration, otherwise it's a fairly clear message.
They also do not sent POST / updates to my /f/inbox instance anymore. Lemmy.ml seems to activity ban Kbin instances. Which is very bad. I really hope this is not on purpose..
I think I had similar issues with one of the mobile clients using my Beehaw account, whenever I tried to write a reply I'd get a 40X (not sure which one exactly) error. I didn't pay attention to the server the post was on though but Beehaw being still on the .17 version might mess with other instances & clients in weird ways.
It's possibly an overreaction to the bot problem.
People started defederating suspicious instances, but what exactly count as suspicious is up for interpretation.
It's also very possible that was the point of the bot signups all along.
And on a relatively unrelated note, splitting the already minuscule fediverse, will kill any chance for it to become a real alternative.
Even if Lemmy's dev political views can be outrageous to some people, defederating them would be the beginning of the end imo.
The issue is, it's only "kbinBot" and variations of that that don't work. If you just use "bot" itself, everything works fine. So it's not just an overreaction to the bot problem, unless all bot instances impersonated kbin.
The more likely answer is that lemmy.ml wanted to exclude kbin users from participating, but without actually limiting the freedom of their own users (which defederation would do).
And on a relatively unrelated note, splitting the already minuscule fediverse, will kill any chance for it to become a real alternative.
Even if Lemmy's dev political views can be outrageous to some people, defederating them would be the beginning of the end imo.
The current situation is misleadingly confusing to the average user and kbin moderators can't do their job properly.
I don't see how defederation, especially when temporary until the issue is fixed, harms anyone when we already can't interact properly anyway. I think the current situation is more harmful. Imo this sounds like the kind of situation defederation exists for.
defederation, especially when temporary
It will create ill will. Perhaps first have kbin's instances admins try to contact the dev.
And if they're unresponsive after a reasonable time then assume bad intent. Or maybe call them out publicly?
If Lemmy's dev them are actually trying to shadowban kbin, yeah fuck em, it would be evil behaviour.
But that would be a serious setback since lemmy has metcalfe's law running for it.
I think dissolution is one of the most obvious and dangerous vectors of attack on the fediverse (along with flooding and the classic embrace and extinguish), so we should keep that in mind.
Not to let paranoia take the better of me, but there are huge financial and political interests rooting against a free, transparent social media platform.
Outraged feelings aren't the problem. Nobody is complaining about defederating Nazis. Defending and lying about Stalinist atrocities is morally no different from defending and lying about Nazi atrocities. If you let Nazis and tankies overrun your communities they won't be a "real alternative." Just like with Nazis, tolerating intolerance is intolerant.
You don't have to go as far as Stalin. The main lemmy dev has Dessalines as an handle.
I understand where you're coming from, but that uncompromising stance will probably mean we'll get corp owned platforms and that will be the end of it.
If I'm proven wrong on that ground, I'll apologize, but I genuinely doubt I will.
We would stand a chance if we made the platform neutral and let users decide whether and how they want to interact with assholes.