update: Beehaw's update downtime was actually just now. oops.
we accidentally did the upgrade just now, which is what you just experienced. anyways we're at like double the power and processing, and three times the storage we were previously at so yeah hopefully that'll be good. anyways i guess i'll leave this post stickied until i go to sleep in like an hour
oh god hopefully not with what we're running now lol, we'll see on the 12th i guess but for some perspective we can probably handle traffic of three to five times the intensity of what we did today (eyeballing it) based on just the metrics i can quickly reference now (i am not our tech person)
Would it be worth adding a CDN (eg cloudflare or fastly) as a preventative measure? I don't know what your traffic distribution is between static and dynamic content, but i imagine being able to offload image GETs would at least prevent you from getting surprise egress traffic bills.
CF also has an "always online" feature to serve a cached version of the site of the server is down. It won't allow people to post it comment, but it might provide a smoother user experience.
I'm primarily watching from the sidelines and able to test things if needed for you guys, but I can definitely say this sudden influx is such a rare opportunity for us tech folks (SWEs and sysadmins) to get a real look at what breaks when you do a real load/soak test of a service.
Post-mortems are amazing for tech knowledge, so seeing it kind of "live" is even better.
If you don't mind my asking, what hardware were y'all using beforehand? I'm looking to host my own instance, but starting as a noob. I know how to build computers but I'm trying to go further.
Beehaw is hosted on DigitalOcean, not sure what price range at this point as I know it's been upgraded recently. It's really easy to spin up one of their "droplets" and get started on a project.
I just got a 14$ droplet with 1 vCPU and 2 gigs of ram with weekly backups. I like their interface and have been happy with it overall so far. Hoping to start up my own lemmy instance for learning purposes, though a bit daunting as apparently the docker instructions are incomplete.
Using Docker is not much harder, actually, at least if you go the docker compose route. You download the docker-compose.yml file, the example config file, and the nginx config file, fill in your domain in the right places, and run docker compose up -d.
The database will provision itself and Lemmy will run all the migrations. Caveat: don't be a smartass like me, the migrations need the Lemmy user to be a superuser so if you use an existing database, you need to either grant temporary superuser permissions or you need to do some manual migrating.
You need an HTTPS certificate, but if you follow the guide that can be set up in as little as two commands (sudo snap install --classic certbot, sudo certbot -d yourdomain.com).
The only problem I ran into was that somehow my admin account wasn't activated. I had to manually enable it (docker compose exec -ti postgres psql, update local_user set accepted_application = true).
If you want a challenge, perhaps consider installing Kbin instead. It mostly interoperates with Lemmy and Mastodon (though it uses boosts ("retweets") rather than likes for upvotes so you may annoy Mastodon people) and it's more manual work to set up, has more optional features to explore (such as S3 file hosting, which you can combine with self-hosted S3 like Minio if you want even more of a learning experience). It also looks better in my opinion.
As for the droplet: I don't know your requirements, but I'm hosting most of my stuff on Contabo because it's a lot cheaper. I probably wouldn't host a business site on there, but it's perfect for hobby projects. You can also try Oracle's free tier*, which has a slightly annoying bandwidth cap of 50mbps but gives you a free/several free (ARM) server(s) with up to 24GB of RAM.
*: Oracle is a terrible company to do business with, make sure not to buy anything from them. After your first month you'll get a phone call from a marketeer, your special cloud credits disappear, and you can keep using your free servers indefinitely.
Congrats on the upgrade! Funnily enough, I decided to finally check if my registration was approved, managed to log in, and immediately wasn't able view the stickied post talking about Beehaw's vision. Glad I didn't kill the site like I thought!
I just went and put some laundry in and I guess it happened then. I remember a time when it would take that long, or longer just to load a single image on a webpage :P