Poor performance and sporadic downtime

Hello!

Apologies for the lack of updates. I just wanted to put up a quick message about this instance. Over the past month or so, there have been times where the site appeared either slow or altogether unresponsive.

Lemmy itself was always running but external monitoring tools would report brief outages and clients would be unable to connect. Usually only for a few seconds at a time.

This ended up being caused by stale DNS records. The infrastructure eviltoast.org is hosted on has multiple public-facing servers which act as load balancers for eviltoast and several other services. Occasionally the load balancers are moved to other servers and the DNS that points to them should get updated automatically.

At some point in the last month or so, the script that updates the DNS added the new records but never deleted the old ones. So depending on which server was used, the instance could appear unresponsive.

This script is fixed now and I tested a couple updates to make sure the same issue doesn't happen again. I'll work on improving the external monitoring and maybe setting up a public status page as well.

For reference, all of these red lines are brief "outages" and you can see when the issue was fixed.

Due to personal reasons (getting married and going on vacation), I didn't really have time to dig into this until recently, so sorry for the delay in getting it fixed and hopefully it didn't cause too much frustration for anyone.