Are there any Discord servers or somewhere in the Matrix to chat about hosting a Lemmy instance? I've got Lemmy running, but I think there are several of us in the same boat struggling with federation performance issues and it might be good to have some place to chat real time.
This is exactly what I am seeing. I just tried upping federation_worker_count in the postgres database. I saw someone in another thread mention trying that so we’ll see.
That guy was me, and it seemed like it worked. Those errors were flooding in, and when I changed the workers to 1024+, they practically stopped except every few seconds, which may not even be my server's fault.
There is an nginx setting you can tune as well. I believe it was worker threads? Can’t remember the exact one and too tired to ssh into my instance to check.
Honestly- a lot of the performance issues aren't due to OUR servers- but, the upstream servers.
beehaw.org, lemmy.world, for example- I think their servers are completely overloaded, and are having issues keeping up.
I don't have sync issues for the smaller/other servers at all. Just the big ones.
I have 128G of ram, 32 cores dedicated. I have the federation worker count set at 256. There is NO shortage of resources, and my server sits more or less, idle.
Due to this only really impacting those larger instances- I believe the blame may lie there.
I think it is less about pointing fingers as to who's at blame, and trying to see if there are things we can do to resolve/alleviate that.
I recall reading somewhere that @[email protected] mentioned before that the server is scaled all the way up to a fairly beefy dedicated server already, perhaps it is soon time to scale this service horizontally across multiple servers. If nothing else, I think a lot of value could be gained by moving the database to a separate server than the UI/backend server as a first step, which shouldn't take too much effort (other than recurring $ and a bit of admin time) even with the current Lemmy code base/deployment workflow...
The UI/Frontend, for example, you can run multiple instances easily.
The API/MiddleTier, I don't know if it supports horizontal scaling though. But, a beefy server can push a TON of traffic.
The database/backend, being postgres, does support some horizontal scaling.
Regarding the app itself, it scales much better if EVERYONE didn't just flock to lemmy.ml, lemmy.world, and beehaw.org. I think that is one of the huge issues.... everyone wanted to join the "big" instance.
Agree on this. When i run docker-compose up and dont detach it, my instance is just constantly asking for updates, and I only get warnings from the biguns.
Honestly I am not sure how scalable this is. It would borderline make more sense to federate authentication and just dump you to the destination instance/community when you click a link and interact directly with it rather than relying on async at the server level.
Yeah, I’ve been selfhosting for nearly a decade and setting up lemmy was, surprisingly, a challenge, and not because it was all that difficult but because the documentation was contradictory, out-of-date, or non-existent in key areas. Federation is my current hurdle, too. It would be nice to have a place to compare notes. Maybe here?
"Also ensure that the time is accurately set on your server. Activities are signed with a timestamp, and will be discarded if it is off by more than 10 seconds."
I was able to adapt the docker compose manifest into a nomad job(yay high availability), but I am really struggling with federation. I have a domain/proper ssl certificate, accessible remotely everything seems OK, but when I try to subscribe to other instances, I get an initial load of posts, then it’s just stuck in subscribe pending. Any time I try to subscribe I see this log message which isn’t exactly helpful about what to do about it…
‘ 2023-06-19T20:11:18.426743Z INFO Worker{worker.id=06aa9ebe-1cab-42fb-ac4b-54bbe7954ba2 worker.queue=default worker.operation.id=fe75d47d-f50d-43d6-921f-795aa50a1b68 worker.operation.name=process}:Job{execution_id=83235752-79dd-4e42-a6f5-d6e32c2e95a9 job.id=ed8bcdbd-4e78-464e-9ae0-871f3d79fd92 job.name=SendActivityTask}: activitypub_federation::core::activity_queue: Target server https://lemmy.ca/inbox rejected https://lemmy.my-domain-redacted.ca/activities/follow/c4b74591-767e-42a0-a160-5023e67c77aa, aborting’
FWIW i see that too on several instances. I dont think it affects anything but syncing with busier instances is a struggle and the destination instance is not acknowledging you following it.
For example if you look at my subscribed you will see the following
But when i go to that page i get the following
And when i click to go the the instance page I get a 500 error, then it works on refresh. And my comments are clearly struggling to stay in sync.
TL:DR - I think some of the larger instances are overloaded its not just user counts and traffic, but also all the backend requests to sync with other instances which may be silently failing/timing out.
OK I was able to resolve this issue on my end, I had setup an internal consul based url during setup and after changing the config it looks like the old url was still lingering in config/db in a bunch of the json responses from the test urls in the documentation. After dropping everything from postgres and re-initializing I seem to be as far as everyone else (some can subscribe, others stuck pending, and no comments :D )
I would be up for something like this. I host my own 8nstance as well. I'm having issues updating communities though. Every time I try I get the button spinner of death. I think in the end, the software is buggy and needs some time to get the bugs worked out, but it is frustrating.