2025-03-15 Unplanned lemm.ee downtime
2025-03-15 Unplanned lemm.ee downtime
Hey folks!
For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.
I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.
Sorry for the issues and I hope you have a nice weekend ahead!
Happy Birthday! I’m using Voyager and experienced some issues and downtime as well. Sneaky bots.
Happy Birthday!! I hope it was a good one!
Thanks for the work, happy birthday!
Happy birthday!
Happy birthday! 🎂
Have a happy rest of your birthday sunaurus!
Thank you! Happy birthday!
We've been dealing with these too at lemmy.ml, don't sweat it, we'll figure it out. Happy bday! Hope it was a good one.
Suspicious timing considering that we have had a large reddit influx.
Happy birthday!
It seems this borked access for third party apps and the different front ends for Lemmy webpage. It is just showing network error
Happy birthday! And thanks for working on that so quickly especially on your birthday.
Happy birthday!
I'm curious about steps you took in reaction to this, of course if sharing compromises the mitigation then by all means, you don't have to
and/or also how do you learn about managing attacks like these ?
No problem, thanks as always for the hard work, and belated happy birthday!
Are the bot checks interfering with third party apps? I have problems connecting with voyager but on browser it's fine after completing the bot checks
Same here. My android app and the front ends I use for the webpage can't connect to lemmee still.
Happy Birthday!
Happy birthday. Check out fail2ban.
that's crazy, happy birthday!!!!
🎉birth
...
Day
Happy: the prequel
We saw a brief similar traffic spike approximately an hour ago, but it dropped off pretty quick. It seemed to focus on our mlmym front end and briefly overloaded our backends for a minute.
All the suspicious ips I've checked so far, are owned by chinese cloud providers (huawei, alibaba, tencent, etc)
Happy birthday! Thanks for all your hard work.
Happy birthday!!! <3
Happy birthday!
Happy birthday! Thanks for your service and hope you enjoy your day.
Happy birthday!
Happy birthday! No big deal at all!
Thank you for taking time out of your birthday to get lemm.ee back online. I hope the rest of your day goes great. Happy Birthday!
Thank you for being born!
your birthday is the ides of march? sick 🤟
Happy birthday
Well, happy birthday! Sucks about this...
I don’t go to this school, but this seems like a particularly strange instance to attack. Were they scrapers? Which is also an incredibly strange thing to do. Spinning up an instance and directly pulling 6 years of content would be far easier.
I read somewhere that the perpetuators are likely getting some practice in before attacking their actual targets. Lemmy instances just happened to be unlucky enough to be among the victims chosen for this rehearsal.
Huh, I hope it wasn't me with my constant Summit (Lemmy client) refreshes (I was having issues with that app so I was doing a lot of API requests I guess) 🤣
Happy birthday! 🥳
Nice work! But remember, there are other instances, and as far as I know your users enjoy your services for free. Take it easy along the way.
Happy birthday!